File: //proc/thread-self/root/usr/share/doc/varnish-6.0.3/html/users-guide/troubleshooting.html
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="X-UA-Compatible" content="IE=Edge" />
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Troubleshooting Varnish — Varnish version 6.0.3 documentation</title>
<link rel="stylesheet" href="../_static/classic.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
<script type="text/javascript" src="../_static/jquery.js"></script>
<script type="text/javascript" src="../_static/underscore.js"></script>
<script type="text/javascript" src="../_static/doctools.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="The Varnish Reference Manual" href="../reference/index.html" />
<link rel="prev" title="Content composition with Edge Side Includes" href="esi.html" />
</head><body>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="../reference/index.html" title="The Varnish Reference Manual"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="esi.html" title="Content composition with Edge Side Includes"
accesskey="P">previous</a> |</li>
<li class="nav-item nav-item-0"><a href="../index.html">Varnish version 6.0.3 documentation</a> »</li>
<li class="nav-item nav-item-1"><a href="index.html" accesskey="U">The Varnish Users Guide</a> »</li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<div class="section" id="troubleshooting-varnish">
<span id="users-trouble"></span><h1>Troubleshooting Varnish<a class="headerlink" href="#troubleshooting-varnish" title="Permalink to this headline">¶</a></h1>
<p>Sometimes Varnish misbehaves or rather behaves the way you told it to
behave but not necessarily the way you want it to behave. In order for
you to understand whats going on there are a couple of places you can
check. <a class="reference internal" href="../reference/varnishlog.html#varnishlog-1"><span class="std std-ref">varnishlog</span></a>, <cite>/var/log/syslog</cite>, <cite>/var/log/messages</cite>
are all good places where Varnish might leave clues of whats going
on. This section will guide you through basic troubleshooting in
Varnish.</p>
<div class="section" id="when-varnish-won-t-start">
<h2>When Varnish won’t start<a class="headerlink" href="#when-varnish-won-t-start" title="Permalink to this headline">¶</a></h2>
<p>Sometimes Varnish wont start. There is a plethora of possible reasons why
Varnish wont start on your machine. We’ve seen everything from wrong
permissions on <cite>/dev/null</cite> to other processes blocking the ports.</p>
<p>Starting Varnish in debug mode to see what is going on.</p>
<p>Try to start Varnish by:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1"># varnishd -f /usr/local/etc/varnish/default.vcl -s malloc,1G -T 127.0.0.1: 2000 -a 0.0.0.0:8080 -d</span>
</pre></div>
</div>
<p>Notice the ‘-d’ parameter. It will give you some more information on what
is going on. Let us see how Varnish will react when something else is
listening on its port.:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1"># varnishd -n foo -f /usr/local/etc/varnish/default.vcl -s malloc,1G -T 127.0.0.1:2000 -a 0.0.0.0:8080 -d</span>
<span class="n">storage_malloc</span><span class="p">:</span> <span class="nb">max</span> <span class="n">size</span> <span class="mi">1024</span> <span class="n">MB</span><span class="o">.</span>
<span class="n">Using</span> <span class="n">old</span> <span class="n">SHMFILE</span>
<span class="n">Platform</span><span class="p">:</span> <span class="n">Linux</span><span class="p">,</span><span class="mf">2.6</span><span class="o">.</span><span class="mi">32</span><span class="o">-</span><span class="mi">21</span><span class="o">-</span><span class="n">generic</span><span class="p">,</span><span class="n">i686</span><span class="p">,</span><span class="o">-</span><span class="n">smalloc</span><span class="p">,</span><span class="o">-</span><span class="n">hcritbit</span>
<span class="mi">200</span> <span class="mi">193</span>
<span class="o">-----------------------------</span>
<span class="n">Varnish</span> <span class="n">Cache</span> <span class="n">CLI</span><span class="o">.</span>
<span class="o">-----------------------------</span>
<span class="n">Type</span> <span class="s1">'help'</span> <span class="k">for</span> <span class="n">command</span> <span class="nb">list</span><span class="o">.</span>
<span class="n">Type</span> <span class="s1">'quit'</span> <span class="n">to</span> <span class="n">close</span> <span class="n">CLI</span> <span class="n">session</span><span class="o">.</span>
<span class="n">Type</span> <span class="s1">'start'</span> <span class="n">to</span> <span class="n">launch</span> <span class="n">worker</span> <span class="n">process</span><span class="o">.</span>
</pre></div>
</div>
<p>Now Varnish is running but only the master process is running, in debug
mode the cache does not start. Now you’re on the console. You can
instruct the master process to start the cache by issuing “start”.:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">start</span>
<span class="n">bind</span><span class="p">():</span> <span class="n">Address</span> <span class="n">already</span> <span class="ow">in</span> <span class="n">use</span>
<span class="mi">300</span> <span class="mi">22</span>
<span class="n">Could</span> <span class="ow">not</span> <span class="nb">open</span> <span class="n">sockets</span>
</pre></div>
</div>
<p>And here we have our problem. Something else is bound to the HTTP port
of Varnish. If this doesn’t help try <code class="docutils literal notranslate"><span class="pre">strace</span></code> or <code class="docutils literal notranslate"><span class="pre">truss</span></code> or come find us
on IRC.</p>
</div>
<div class="section" id="varnish-is-crashing-panics">
<h2>Varnish is crashing - panics<a class="headerlink" href="#varnish-is-crashing-panics" title="Permalink to this headline">¶</a></h2>
<p>When Varnish goes bust the child processes crashes. Most of the
crashes are caught by one of the many consistency checks we have included in the Varnish source code. When Varnish hits one of these the caching
process will crash itself in a controlled manner, leaving a nice
stack trace with the mother process.</p>
<p>You can inspect any panic messages by typing <code class="docutils literal notranslate"><span class="pre">panic.show</span></code> in the CLI.:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">panic</span><span class="o">.</span><span class="n">show</span>
<span class="n">Last</span> <span class="n">panic</span> <span class="n">at</span><span class="p">:</span> <span class="n">Tue</span><span class="p">,</span> <span class="mi">15</span> <span class="n">Mar</span> <span class="mi">2011</span> <span class="mi">13</span><span class="p">:</span><span class="mi">09</span><span class="p">:</span><span class="mi">05</span> <span class="n">GMT</span>
<span class="n">Assert</span> <span class="n">error</span> <span class="ow">in</span> <span class="n">ESI_Deliver</span><span class="p">(),</span> <span class="n">cache_esi_deliver</span><span class="o">.</span><span class="n">c</span> <span class="n">line</span> <span class="mi">354</span><span class="p">:</span>
<span class="n">Condition</span><span class="p">(</span><span class="n">i</span> <span class="o">==</span> <span class="n">Z_OK</span> <span class="o">||</span> <span class="n">i</span> <span class="o">==</span> <span class="n">Z_STREAM_END</span><span class="p">)</span> <span class="ow">not</span> <span class="n">true</span><span class="o">.</span>
<span class="n">thread</span> <span class="o">=</span> <span class="p">(</span><span class="n">cache</span><span class="o">-</span><span class="n">worker</span><span class="p">)</span>
<span class="n">ident</span> <span class="o">=</span> <span class="n">Linux</span><span class="p">,</span><span class="mf">2.6</span><span class="o">.</span><span class="mi">32</span><span class="o">-</span><span class="mi">28</span><span class="o">-</span><span class="n">generic</span><span class="p">,</span><span class="n">x86_64</span><span class="p">,</span><span class="o">-</span><span class="n">sfile</span><span class="p">,</span><span class="o">-</span><span class="n">smalloc</span><span class="p">,</span><span class="o">-</span><span class="n">hcritbit</span><span class="p">,</span><span class="n">epoll</span>
<span class="n">Backtrace</span><span class="p">:</span>
<span class="mh">0x42cbe8</span><span class="p">:</span> <span class="n">pan_ic</span><span class="o">+</span><span class="n">b8</span>
<span class="mh">0x41f778</span><span class="p">:</span> <span class="n">ESI_Deliver</span><span class="o">+</span><span class="mi">438</span>
<span class="mh">0x42f838</span><span class="p">:</span> <span class="n">RES_WriteObj</span><span class="o">+</span><span class="mi">248</span>
<span class="mh">0x416a70</span><span class="p">:</span> <span class="n">cnt_deliver</span><span class="o">+</span><span class="mi">230</span>
<span class="mh">0x4178fd</span><span class="p">:</span> <span class="n">CNT_Session</span><span class="o">+</span><span class="mi">31</span><span class="n">d</span>
<span class="p">(</span><span class="o">..</span><span class="p">)</span>
</pre></div>
</div>
<p>The crash might be due to misconfiguration or a bug. If you suspect it
is a bug you can use the output in a bug report, see the “Trouble Tickets” section in the Introduction chapter above.</p>
</div>
<div class="section" id="varnish-is-crashing-segfaults">
<h2>Varnish is crashing - segfaults<a class="headerlink" href="#varnish-is-crashing-segfaults" title="Permalink to this headline">¶</a></h2>
<p>Sometimes a bug escapes the consistency checks and Varnish gets hit
with a segmentation error. When this happens with the child process it
is logged, the core is dumped and the child process starts up again.</p>
<p>A core dumped is usually due to a bug in Varnish. However, in order to
debug a segfault the developers need you to provide a fair bit of
data.</p>
<blockquote>
<div><ul class="simple">
<li>Make sure you have Varnish installed with debugging symbols.</li>
<li>Make sure core dumps are allowed in the parent shell. (<code class="docutils literal notranslate"><span class="pre">ulimit</span> <span class="pre">-c</span> <span class="pre">unlimited</span></code>)</li>
</ul>
</div></blockquote>
<p>Once you have the core you open it with <cite>gdb</cite> and issue the command <code class="docutils literal notranslate"><span class="pre">bt</span></code>
to get a stack trace of the thread that caused the segfault.</p>
</div>
<div class="section" id="varnish-gives-me-guru-meditation">
<h2>Varnish gives me Guru meditation<a class="headerlink" href="#varnish-gives-me-guru-meditation" title="Permalink to this headline">¶</a></h2>
<p>First find the relevant log entries in <a class="reference internal" href="../reference/varnishlog.html#varnishlog-1"><span class="std std-ref">varnishlog</span></a>. That will
probably give you a clue. Since <a class="reference internal" href="../reference/varnishlog.html#varnishlog-1"><span class="std std-ref">varnishlog</span></a> logs a lot of
data it might be hard to track the entries down. You can set
<a class="reference internal" href="../reference/varnishlog.html#varnishlog-1"><span class="std std-ref">varnishlog</span></a> to log all your 503 errors by issuing the
following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ varnishlog -q 'RespStatus == 503' -g request
</pre></div>
</div>
<p>If the error happened just a short time ago the transaction might
still be in the shared memory log segment. To get <a class="reference internal" href="../reference/varnishlog.html#varnishlog-1"><span class="std std-ref">varnishlog</span></a>
to process the whole shared memory log just add the ‘-d’ parameter:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ varnishlog -d -q 'RespStatus == 503' -g request
</pre></div>
</div>
<p>Please see the <a class="reference internal" href="../reference/vsl-query.html#vsl-query-7"><span class="std std-ref">vsl-query</span></a> and <a class="reference internal" href="../reference/varnishlog.html#varnishlog-1"><span class="std std-ref">varnishlog</span></a> man pages
for elaborations on further filtering capabilities and explanation of
the various options.</p>
</div>
<div class="section" id="varnish-doesn-t-cache">
<h2>Varnish doesn’t cache<a class="headerlink" href="#varnish-doesn-t-cache" title="Permalink to this headline">¶</a></h2>
<p>See <a class="reference internal" href="increasing-your-hitrate.html#users-guide-increasing-your-hitrate"><span class="std std-ref">Achieving a high hitrate</span></a>.</p>
</div>
</div>
</div>
</div>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
<div class="sphinxsidebarwrapper">
<h3><a href="../index.html">Table Of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">Troubleshooting Varnish</a><ul>
<li><a class="reference internal" href="#when-varnish-won-t-start">When Varnish won’t start</a></li>
<li><a class="reference internal" href="#varnish-is-crashing-panics">Varnish is crashing - panics</a></li>
<li><a class="reference internal" href="#varnish-is-crashing-segfaults">Varnish is crashing - segfaults</a></li>
<li><a class="reference internal" href="#varnish-gives-me-guru-meditation">Varnish gives me Guru meditation</a></li>
<li><a class="reference internal" href="#varnish-doesn-t-cache">Varnish doesn’t cache</a></li>
</ul>
</li>
</ul>
<h4>Previous topic</h4>
<p class="topless"><a href="esi.html"
title="previous chapter">Content composition with Edge Side Includes</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="../reference/index.html"
title="next chapter">The Varnish Reference Manual</a></p>
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="../_sources/users-guide/troubleshooting.rst.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
<div id="searchbox" style="display: none" role="search">
<h3>Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="../search.html" method="get">
<input type="text" name="q" />
<input type="submit" value="Go" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../genindex.html" title="General Index"
>index</a></li>
<li class="right" >
<a href="../reference/index.html" title="The Varnish Reference Manual"
>next</a> |</li>
<li class="right" >
<a href="esi.html" title="Content composition with Edge Side Includes"
>previous</a> |</li>
<li class="nav-item nav-item-0"><a href="../index.html">Varnish version 6.0.3 documentation</a> »</li>
<li class="nav-item nav-item-1"><a href="index.html" >The Varnish Users Guide</a> »</li>
</ul>
</div>
<div class="footer" role="contentinfo">
© Copyright 2010-2014, Varnish Software AS.
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.7.6.
</div>
</body>
</html>