CrossRoads Upgrade / Crash and some misc questions :D

Help forum for installing and using Crossroads.

CrossRoads Upgrade / Crash and some misc questions :D

Postby kristiaan_d » Thu Jun 03, 2010 5:09 pm

Hi Karel,
its been a while since we last spoke which to me is a good testiment to how reliable CrossRoads, however i do have a minor problem that popped up this morning.

Basically we have a linux box that is running fedora this is running XR 2.55 which serves as a load balancer for us for HTTP and HTTPS (dont panic i will post configs and logs shortly just want to explain whats going on first).

for no reason i can find other than one entry in a log file both of the XR services fell over today and refused to restart no matter what i tried, the entry i found in the log file is as follows.

Code: Select all
2834385808 ERROR: Write/send failed: errno=104, Connection reset by peer, result=-1


not sure if this is the reason for the crash earlier today or not but this is all that i could find, there was no hint to a problem in /usr/log/messages and all previous files in there were also clear of XR errors.


here is our config file for XR (sensitive info removed).

Code: Select all
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <system>
    <uselogger>true</uselogger>
    <logdir>/tmp</logdir>
  </system>

  <service>
    <name>web_http_IP_167</name>
    <server>
      <address>INTERNAL_IP:80</address>
      <type>tcp</type>
      <dispatchmode>first-available</dispatchmode>
      <clienttimeout>60:60</clienttimeout>
      <backendtimeout>60:60</backendtimeout>
    </server>
      <backend>
        <address>62.254.243.69:80</address>
      </backend>
      <backend>
        <address>89.151.84.163:80</address>
      </backend>
      <backend>
        <address>127.0.0.1:81</address>
      </backend>
  </service>

  <service>
    <name>web_ssl_IP_167</name>
    <server>
      <address>INTERNAL_IP:443</address>
      <type>tcp</type>
      <dispatchmode>first-available</dispatchmode>
      <clienttimeout>60:60</clienttimeout>
      <backendtimeout>60:60</backendtimeout>
   </server>
    <backend>
      <address>62.254.243.69:443</address>
    </backend>
    <backend>
      <address>89.151.84.163:443</address>
    </backend>
    <backend>
      <address>127.0.0.1:81</address>
    </backend>
  </service>
<configuration>


here is a copy of what gets generated from XR -V

Code: Select all
XR version    : 2.55
Written by    : Karel Kubat <karel@kubat.nl>
Maintained by : Karel Kubat <karel@kubat.nl>
Primary site  : http://crossroads.e-tunity.com
Compiled with : /usr/bin/g++
Optimization  : -O3
System        : Linux
Libraries     : -L/usr/lib -lnsl -lpthread -lm
Type sizes    : ssize_t=4, int=4, long=4, double=8, ptr=4
getopt.h      : present
INADDR_NONE   : present
getopt_long() : present
inet_aton()   : present
strnstr()     : absent


apart from it being an old version of XR I cannot really find a reason for the crash, we ended up having to reboot the linux box before XR would start again.

i was just hoping for some advice and answers to the questions below if possible.


1: how do i go about investigating a crash like within XR for future reference as having no reason for this sort of crash upsets my IT Manager.
2: what's the easiest way to upgrade from the version we are using now to the latest version and do we need to make any config file or server changes to accommodate this upgrade?
3: im still a very basic Linux user so i would appreciate any advice given made as clear as possible :) that way i get the work done in one go and don't have to keep pestering everyone :)
kristiaan_d
 
Posts: 38
Joined: Mon Apr 27, 2009 6:05 pm

Re: CrossRoads Upgrade / Crash and some misc questions :D

Postby Karel » Thu Jun 03, 2010 8:43 pm

Re-hi Christian,

I am glad that all in all XR works for you ;-)

1. The log entry "write error, connection reset by peer" is what many browsers cause. I have taken it out of the error message set, it's just too common. I don't think it's releated to your crash.

2. I am sorry that you needed to reboot to get XR up and running - very odd. Since XR is a userland program, maybe the OS went into a death spriral? Sounds far fetched, but hmmm, interesting problem ..

3. As for upgrading - you can keep your XML config, just grab the latest source (2.65 if memory serves me) and make install. Then restart. I don't see any problems there.

4. Incase it happens again - sorry to say, but there are just too many possibilities. You can always follow these steps:
- Make really sure XR is down, us ps ax to verify.
- Do xrctl -v configtest. It will show you the plain XR command.
- Run that plain command in a term, and watch for output. Add --verbose and --debug if it's not there yet.

Hope this helps - else let me know,
Karel
Karel
Site Admin
 
Posts: 308
Joined: Mon Sep 22, 2008 11:13 am
Location: Netherlands

Re: CrossRoads Upgrade / Crash and some misc questions :D

Postby kristiaan_d » Mon Jul 26, 2010 10:52 am

Hi Karel,
thanks for you reply regarding the problems i was getting, unfortunatly since we last spoke XR has crashed twice more. oddly i checked the tmp folder and viewed the log files and again the only entry there was

Code: Select all
2970647440 ERROR: Write/send failed: errno=104, Connection reset by peer, result=-1


however i did not need to restart the server this time a simple XRCTL START <servicename> got them both running again as though nothing had happened.
My Config file has not changed since i last posted it (heres a copy again just in case).

Is there anything sort of setting or command i can enter into the config file to provide better log details?

config:
Code: Select all
    <?xml version="1.0" encoding="UTF-8"?>
    <configuration>
      <system>
        <uselogger>true</uselogger>
        <logdir>/tmp</logdir>
      </system>

      <service>
        <name>web_http_IP_167</name>
        <server>
          <address>INTERNAL_IP:80</address>
          <type>tcp</type>
          <dispatchmode>first-available</dispatchmode>
          <clienttimeout>60:60</clienttimeout>
          <backendtimeout>60:60</backendtimeout>
        </server>
          <backend>
            <address>62.254.243.69:80</address>
          </backend>
          <backend>
            <address>89.151.84.163:80</address>
          </backend>
          <backend>
            <address>127.0.0.1:81</address>
          </backend>
      </service>

      <service>
        <name>web_ssl_IP_167</name>
        <server>
          <address>INTERNAL_IP:443</address>
          <type>tcp</type>
          <dispatchmode>first-available</dispatchmode>
          <clienttimeout>60:60</clienttimeout>
          <backendtimeout>60:60</backendtimeout>
       </server>
        <backend>
          <address>62.254.243.69:443</address>
        </backend>
        <backend>
          <address>89.151.84.163:443</address>
        </backend>
        <backend>
          <address>127.0.0.1:81</address>
        </backend>
      </service>
    <configuration>
kristiaan_d
 
Posts: 38
Joined: Mon Apr 27, 2009 6:05 pm

Re: CrossRoads Upgrade / Crash and some misc questions :D

Postby Karel » Mon Jul 26, 2010 2:09 pm

Hi again Kristiaan,

Your config file is just fine, no reason to change it.

If you want to get to the bottom of the crashes, then I fear that the only way is to enable verbosity, enable debugging, wait 'till it crashes, and then look at the logs (I can do that for you if you like).

However, this might not be too fruitful. I'd suggest you upgrade to the latest version, which really has some bugfixes. Then, if the crashes still occur, let's dig into it. But I think that upgrading might really fix it without the need to go into deep-debug mode.

/Karel
Karel
Site Admin
 
Posts: 308
Joined: Mon Sep 22, 2008 11:13 am
Location: Netherlands

Re: CrossRoads Upgrade / Crash and some misc questions :D

Postby kristiaan_d » Mon Jul 26, 2010 2:30 pm

Hi Karel,
Firstly apologies i neglected to mention the XR details from my last email, so there posted below, i run an upgrade on the XR software a while back when we first encountered this crash as i seen some information on here about it resolving problems that may cause crashes.

Code: Select all
XR version    : 2.65
Written by    : Karel Kubat <karel@kubat.nl>
Maintained by : Karel Kubat <karel@kubat.nl>
Primary site  : http://crossroads.e-tunity.com
Compiled with : /usr/lib/ccache/g++
Optimization  : -O3
System        : Linux
Libraries     : -L/usr/lib -lnsl -lpthread -lm
Type sizes    : ssize_t=4, int=4, long=4, double=8, ptr=4
getopt.h      : present
INADDR_NONE   : present
getopt_long() : present
inet_aton()   : present
strnstr()     : absent


i have XR running on two separate (but identical, ones a clone of the other) Linux boxes doing the same job (only difference is the IP Addresses) so far the one is stable and happy with no signs of random or intermittent crashes, however this one has crashed at least three times so far.
kristiaan_d
 
Posts: 38
Joined: Mon Apr 27, 2009 6:05 pm

Re: CrossRoads Upgrade / Crash and some misc questions :D

Postby Karel » Mon Jul 26, 2010 2:32 pm

Hi again,

that *is* weird, why should one crash, and one not?

I am afraid that the only thing left to do is: enable verbosity, enable debug logging, run it and catch the event in the logs. Then I'll have a look at it.

/Karel
Karel
Site Admin
 
Posts: 308
Joined: Mon Sep 22, 2008 11:13 am
Location: Netherlands

Re: CrossRoads Upgrade / Crash and some misc questions :D

Postby kristiaan_d » Mon Jul 26, 2010 2:46 pm

Hi Karel,
no problems i will get this arranged as soon as i can, however been as these are production systems for our company it may take a little while to get the OK to-do this and get it setup and working.

with regards to the setup so i take it to enable debug and verbose modes for XR on this PC i just do

Code: Select all
XRCTL START <MYSERVICE> --debug --verbose


or is there another way of doing it? also i am guessing that been as the Config file states the log folder is /tmp that it will create a file in here for the name of the service?

Thanks
Kris
kristiaan_d
 
Posts: 38
Joined: Mon Apr 27, 2009 6:05 pm

Re: CrossRoads Upgrade / Crash and some misc questions :D

Postby Karel » Wed Jul 28, 2010 10:58 am

Kris,

Just go to the web interface and toggle debug and verbose. Given your configuration I believe that the output goes to syslog (so /var/log/messages probably).

/Karel
Karel
Site Admin
 
Posts: 308
Joined: Mon Sep 22, 2008 11:13 am
Location: Netherlands

Re: CrossRoads Upgrade / Crash and some misc questions :D

Postby kristiaan_d » Wed Jul 28, 2010 5:41 pm

Karel wrote:Kris,

Just go to the web interface and toggle debug and verbose. Given your configuration I believe that the output goes to syslog (so /var/log/messages probably).

/Karel



Hi Karel,
at present i do not have the web interfaces running for the Load Balancers, how do i go about enabling them so i can get this option setup.
kristiaan_d
 
Posts: 38
Joined: Mon Apr 27, 2009 6:05 pm

Re: CrossRoads Upgrade / Crash and some misc questions :D

Postby Karel » Wed Jul 28, 2010 11:51 pm

Hi Kris,

You enable the web interface by specifying it inside the <server> block, in your case:
Code: Select all
<server>
          <webinterface>INTERNAL_IP:8001</webinterface>
          <address>INTERNAL_IP:80</address>
          <type>tcp</type>
          <dispatchmode>first-available</dispatchmode>
          <clienttimeout>60:60</clienttimeout>
          <backendtimeout>60:60</backendtimeout>
</server>


Then surf to http://INTERNAL_IP:8001. Good luck!
Karel
Site Admin
 
Posts: 308
Joined: Mon Sep 22, 2008 11:13 am
Location: Netherlands

Re: CrossRoads Upgrade / Crash and some misc questions :D

Postby kristiaan_d » Tue Aug 31, 2010 12:36 pm

Hi Karel,
just a quick update for you on our intermittent problems.

after enabling the web interface i switched on Debug Logging and verbose logging. and have sat back and waited.

one of the xr instances crashed today and i got it restarted, the initial XRCTL STATUS request showed that one was running and one was not.

Code: Select all
[root@ip-XXXXXX ~]# xrctl status
Service web_http_IP_164: running
Service web_ssl_IP_164: not running


so i did

Code: Select all
[root@ip-XXXXXX ~]# xrctl start web_ssl_IP_164
Service web_ssl_IP_164: started


according to my config i am using /tmp as my log folder but upon checking this folder its empty. i am guessing this is incorrect considering my config states a log folder, but checking the web interface shows that the Traffic Log Directory box is empty, is this likely to be anything to-do with it??


(heres a copy of my config)

Code: Select all
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <system>
    <uselogger>true</uselogger>
    <logdir>/tmp</logdir>
  </system>

  <service>
    <name>web_http_IP_164</name>
    <server>
      <webinterface>XXXXXX:XX</webinterface>
      <address>XXXXXX:80</address>
      <type>tcp</type>
      <dispatchmode>first-available</dispatchmode>
      <clienttimeout>60:60</clienttimeout>
      <backendtimeout>60:60</backendtimeout>
    </server>
      <backend>
        <address>62.254.246.245:80</address>
      </backend>
      <backend>
        <address>89.151.84.164:80</address>
      </backend>
      <backend>
        <address>XXXXXX</address>
      </backend>
  </service>

  <service>
    <name>web_ssl_IP_164</name>
    <server>
      <address>XXXXXX:443</address>
      <type>tcp</type>
      <dispatchmode>first-available</dispatchmode>
      <clienttimeout>60:60</clienttimeout>
      <backendtimeout>60:60</backendtimeout>
   </server>
    <backend>
      <address>62.254.246.245:443</address>
    </backend>
    <backend>
      <address>89.151.84.164:443</address>
    </backend>
    <backend>
      <address>XXXXXX</address>
    </backend>
  </service>
<configuration>


heres a copy of XR -V

Code: Select all
[root@ip-XXXXXX tmp]# xr -V
XR version    : 2.65
Written by    : Karel Kubat <karel@kubat.nl>
Maintained by : Karel Kubat <karel@kubat.nl>
Primary site  : http://crossroads.e-tunity.com
Compiled with : /usr/lib/ccache/g++
Optimization  : -O3
System        : Linux
Libraries     : -L/usr/lib -lnsl -lpthread -lm
Type sizes    : ssize_t=4, int=4, long=4, double=8, ptr=4
getopt.h      : present
INADDR_NONE   : present
getopt_long() : present
inet_aton()   : present
strnstr()     : absent


any suggestions would be appreciated as i am not sure where my logs have gone or where they should be turning up. id really like to stop these crashes if possible as its driving our IT Manager up the wall.

Kris
kristiaan_d
 
Posts: 38
Joined: Mon Apr 27, 2009 6:05 pm

Re: CrossRoads Upgrade / Crash and some misc questions :D

Postby Karel » Wed Sep 01, 2010 12:02 am

Hi again Kris,

I just want to say, I saw your post and I'll see what I can do, though there is no extra info :(
This week I'm swamped in work but I'll see what I can do...

Just one more thing, there is not a system message by chance in /var/log/messages or /var/log/kernel about exhaustion of file descriptors or memory or the such?

/Karel
Karel
Site Admin
 
Posts: 308
Joined: Mon Sep 22, 2008 11:13 am
Location: Netherlands


Return to Crossroads Help

Who is online

Users browsing this forum: No registered users and 1 guest

cron