Allstream monitored the T-1 connection throughout the day yesterday then ran their intensive tests last night between 8pm and 11pm.
When I got the report from them this morning they have changed their story about the fault being with their equipment, they are now saying that it’s a problem on Maksystems’ network.

Essentially the setup is that their router is plugged into a device called a ‘switch’ which then connects to all the servers here.  What they are saying is that the problem is probably with our switch hardware.  I’m not entirely convinced because the problem could still be with their router.

Nonetheless I am taking steps here to further isolate things, and will replace the ‘switch’ device at the earliest opportunity.  After that if the problem continues to occur that will leave only their router as the culprit and I will demand that they replace it.  Until I can prove that it’s not our switch, they won’t do anything.

So unfortunately I can’t state that the problem is resolved yet.  All I can do is ask that you all continue to report any ongoing problems to me, with as much detail as possible.

I will replace the switch around noon today which will result in about 5 minutes of downtime.

Thanks to the some excellent problem reports received this morning, I was able to coordinate with our internet provider Allstream to identify a recurring intermittent issue in their logs.

It seems that there are indeed random, brief connection drops on our T-1 line. They have been brief enough not to raise any flags or alarms here, but should it occur during a data transfer, the transfer is interupted which can result in either a broken / stuttering print job, or a dropped connection to NorthstAr / PolAris.

Other services such as email and web are typically uneffected as they are ‘asynchronous’ in nature, meaning that if a packet is lost in transit, it is not harmful to resend it.  ‘Synchronous’ services however (NorsthAr, PolAris, and print streams) are broken when this sort of connection-drop occurs.

Having said all this, the problem is not yet solved -but at least it has been identified and Allstream is now working on it at their end. They will be closely monitoring the T-1 line throughout the day today, and may be able to catch and isolate the problem that way. If not, then they take Maksystems offline tonight (after 6:00pm) to run some intensive tests on their equipment, in order to identify where exactly the fault lies.

In any event, I will continue to post updates as news becomes available.

It seems that the original intermittent printing problems that the recent server upgrade was intended to fix, are still with us.

The problem that some customers have reported is that print jobs get interupted so that only part of the job is printed — for multi-page jobs, only some pages are printed, for single-page jobs, only part of the page is printed. Then the print job is re-sent, and can be re-sent over and over until it gets through completely.

Initially the problem was only reported at one location, and I believed this to be an internet/communications problem. However when a second client reported the problem, I then suspected it was due to a problem with the printer server itself (hence the upgrade).

With the server eliminated as the source of the trouble though, I am again looking at communications/internet issues as the cause. The intermittent nature of the problem unfortunately makes this a very difficult one to diagnose – some print jobs seem to work, while others do not.

If anyone experiences the problems I have described here, please email as much of the following information as you can:

• What time exactly did you notice the problem?
• What exactly were you printing (i.e. invoice, purchase order, etc.)?
• How many pages were you expecting?
• If you had a hold entry number, what was it?

As I said, the intermittent nature of this problem is making it incredibly difficult to pin down. The more information and the more details that can be compiled, the sooner we will be able to find and fix it.

I am attempting to work with our ISP (Allstream) but for their engineers to help we really need detailed information.

Finally, the function to print Both copies of an invoice at the same time is working. As this was the last outstanding issue, everything seems to be working correctly now with the new Printer Server.

I will of course keep the old one onhand for now, just incase anything else comes up.

I do not expect though that we will need it, as the new hardware and software now seems to be functioning correctly.

Once again, I apologize for any inconvenience that the change-over may have caused.

The Print-to-PDF / Email functions are now performing properly again, so if you select email as your print destination, the email should arrive after a few minutes, as usual.

I am still working on the other issue, where printing “B”oth copies of an invoice is not working.

In an attempt to solve a printing problem that has affected some of our clients over the past few weeks, we are replacing the hardware and software that drives our print (and email) server.

The same box handles both responsibilities, and it has performed fairly well since approximately 2004 when it was installed.

The email server software will not be changing much, and the upgrade will probably have minimal impact.

On the printing side, we’re moving from DocForm version 3.5 to DocForm version 5.1. The hardware upgrade may also improve performance, as we’re moving from an older machine with limited speed and capability to an Intel Core-2 Duo 2.5GHz CPU and 2GB of RAM.