Chris Whisonant Blogs

Some Downtime Wednesday Evening

2009-07-04T18:15:40-00:00

Tomorrow evening (Wednesday, June 24, 2009) we will have some downtime on almost all of the BleedYellow servers. This includes the Connections and Sametime servers. It&os;s a planned, routine maintenance outage to load some fixes, etc...

Thanks!

How Google Killed Old Yeller!

2009-07-04T18:15:40-00:00

As many of you may know, we have had a problem with Dogear here for a while. At one point, I started referring to it as Old Yeller (could not resist the dead dog/yellow reference - sorry). After doing a lot of upgrades (to IHS/WAS/Connections), DB troubleshooting, completely deleting the DB (which has since been restored), enabling advanced Windows logging (which did not help much), even moving to a fresh VM with a clean install of WAS, maintenance, etc..., nothing seemed to work. In all the crashes of the Dogear jvm, only once did a good Dr. Watson dump get sent out. But in one of them we noticed what looked like a strange search of the Dogear database. After further digging, I looked at this closer with IBM. Turns out we were seeing a lot of these in the IHS access.log:

It was not always the same search - it would be for different users, etc.... The IP address, though, is a Googlebot. So, in walks robots.txt to see if he can help! After adding it to the server, I wondered how long it would take for Google to honor the settings as we kept seeing the crashes. As part of the move to a new box with Deployment Manager, the Dogear jvm would automagically restart. So that helped uptime a bit. I noticed that the last crawl of Dogear by Google was around 8:25 PM on 6/12/09. And at 8:29 PM Dogear was last restarted. So it appears that Google took around 12 hours to start honoring the robots.txt.

After running now for almost a week, I think we are confident in saying that Dogear has issues with this type of string in the crawl. After some further data crunching from 6/12/09 (where we logged 7 Dogear crashes between 7:11 pm and 8:30 pm), here are some quick stats on when we saw Dogear get crawled with this string in the IHS access.log and when the server recovery message was logged in the WAS SystemOut.log:

the string "lang=en?ref=sex%" was found in IHS access.log
6/12 19:07
6/12 19:12
6/12 19:25
6/12 19:43
6/12 19:50, 19:52
6/12 20:06
6/12 20:25

Restarts in SystemOut.log
6/12 19:11
6/12 19:15
6/12 19:29
6/12 19:44
6/12 19:57
6/12 20:11
6/12 20:29

Since those times on 6/12, the crawl has not occurred any more and Dogear has not crashed. Here&os;s to hoping this is finally resolved!

A Note for BleedYellow Bloggers

2009-07-04T18:15:40-00:00

To all of our wonderful bloggers here, you are probably aware of the issues we have been working through with Dogear. This has the potential to make your blog appear unresponsive when Dogear hangs before automagically restarting itself. Before it would just die, now it restarts. If you have the "Dogear Bookmarks" available in your blog template, then this is what causes your blog to take a while to load sometimes. If this gets too irritating for you, you can modify your template to simply disable the Dogear integration (at least temporarily until we get this sorted out).

Thanks!

DAOS Confusion!

2009-07-04T18:15:40-00:00

Or: Can I DAOSify a Server With < 8.5 Clients?

Recently I have had some colleagues and clients asking about updating a Domino Server to 8.5 and turning on DAOS while still running a mix of 6 and 7 Notes Clients. There is some confusion related to some upgrade seminars that are happening that appear to be using this slideshow. Here is slide 18 that is causing some confusion:

The last bullet is causing confusion. Of course "Domino 6 and 7 clients will not recognize DAOS" - even the 8.0.x client would fall into this category as it was released pre-DAOS - DAOS is a server feature. However, there are no known problems with using an older client against a DAOSified mail template. Here is why: the Domino Server is aware of DAOS. Any requests for an attachment from clients, servers, or services such as BES will be honored by the Domino server sending the attachment. The server will still have to get the nlo file and serve it to the requesting client (or server) over the wire. And, if you are running local replicas it does not matter either because the replica will be pulled down with all attachments inside the database.

Do I recommend a long-term strategy of having 6 and 7 clients with an 8.5 server? Negative. But you can immediately reap some benefits by upgrading your server and enabling DAOS while you are working on your client rollout strategy. And, as the slide above states, you&os;ll need to wait until the clients are upgraded before you can update the template to mail85.ntf. Finally, DAOS is not just for mail databases. I have also sensed some confusion around this. Of course, in most environments, the mail store is where most databases would have duplicate attachments (either internal or shared) and most of DAOS&os;s storage and i/o savings has been revolved around mail files.

Hopefully this helps!

Re: FTOTM (first Tuesday of the month) = new rules

2009-07-04T18:15:40-00:00

In response to: FTOTM (first Tuesday of the month) = new rules

I never knew that you could actually disable xPages on the server completely. This is rather interesting. What&os;s funnier is that the notes.ini that&os;s used to do this is INOTESDISABLEXPAGECMD - iNotes and xPages are not even tied together at this point.

Update On Dogear

2009-07-04T18:15:40-00:00

As many of you know, we have been having some issues with Dogear here and working with IBM. As we have upgraded, moved, and tried all kinds of different things, it appears to have been pointing to the database as the culprit. For now, I have created a new Dogear db to see if this is the problem. I have a full backup of the db from last night, and if it turns out that the DB was not the problem, then I will restore that backup. Sorry for the inconvenience - we have tried a lot of things for many many months to get this sorted out. Thanks for your continued patience on this. I am hoping that this will not be the issue and that we can restore it back, but after touching all of the other components (upgrades, reinstalls, moves, etc...) and running corrective actions on the db with no luck, this appears to be the next logical step.

When IBM Meets Star Trek

2009-07-04T18:15:40-00:00

Just thought I would remind everyone out there about the webadmin logo that IBM has been using over the years. Funny how close the 7.0 Beta 4 was to the actual Star Trek logo with the starship in the center. It was changed for the 7.0 GOLD release.

So, I wonder what IBM is trying to say about administrators?? By the way, I&os;m hoping to go see the new movie this week!

Updated DAOS Estimator

2009-07-04T18:15:40-00:00

On April 16, 2009, IBM released version 1.4 of the DAOS estimator. I like some of the new options.

They are:

-i option allows the estimator to run on an indirect file which contains a list of databases to run against.
-c options causes the estimator to write attachment data to a CSV file for later processing. This is up to 65% faster to acquire data.
-a option analyzes a CSV file created with the -c option. This option can also be used to analyze a set of CSV files using an &os;.ind&os; file to list the CSV files.
-p option generates an estimate of the estimate by running against a percentage of databases and extrapolating the results.

Something that IBM hasn&os;t pointed out in the technote is that it appears that using the -c option will allow for you to run multiple instances of the Estimator and then offload the analysis to a different server. Consider that you have multiple mail directories. If you run the estimator against each of them using the -c option, this will likely finish faster for acquiring the data. Then, you should be able to copy those CSV files to another box and use the -a -i option to analyze the multiple CSV files without hogging more CPU on your production mail server. Remember, it took Mitch Cohen around 5 days to run the estimator on a big server with 7,000 mail files. So, having that task not perform the analysis on a mail server would definitely help out!

By the way, indirect files are one of the most under-utilized options with Domino. Look into them and start using them!

The 'Tuner Blog

2009-07-04T18:15:40-00:00

Nope, I&os;m not talking about a blog for people into fast and furious 4 bangers (no offense to 4 banger drivers - like myself - who aren&os;t fast or furious).

I&os;m talking about the new blog that IBMer Scott O&os;Keefe has started related to the Domino Configuration Tuner (DCT). You should go check it out at &os;Tuner Blog. The first post is related to the "Check for Updates" option that&os;s available with the template in which you can get the latest template from IBM as well as the latest updates to the rules. He also lists some of the new rules with the latest release.

You should really look into DCT if you haven&os;t already. And with the &os;Tuner Blog now active, I&os;m certain that Scott, et. al. will be looking for feedback!

Re: Concerns over DAOS Estimator?

2009-07-04T18:15:40-00:00

In response to: Concerns over DAOS Estimator?

Just a note - I have seen what Vitor is referring to in his comment on my initial post and have verified that there are cases where if the file name is the same as well as the size reported in the DAOS Estimator report that the files are treated as the same object by the Estimator. But after enabling DAOS on the database and compacting it, the two attachments were treated as separate objects by DAOS even though the tool did not see them as different. This means that you may save less space than the Estimator thinks due to some edge cases where you may possibly have attachments with the same name and physical file size but with different internal data. In Vitor&os;s case with auto-generated files, the results may be very skewed. Again, each environment is different and ymmv. IBM is already aware of these findings above.