Today its nothing special but work,Back to home at 9:20pm.Nothing to post,Im planning to share my DayOne experiences at PHI,Lafayette.
A bit about the PHI application installation & architecture :
PHI Aviation Solution is an 100% J2EE Solution developed n par with the latest J2EE Standards.
Presentation Layer : Static HTML’s/Javascripts/JSP/Servlets/javabeans
Business Layer : Stateless EJB’s & some Stateful beans
Data Layer : Stored procedures/tables
Reports : JReports Enterprise Server 6.1
Weblogic 7.0 sp4 will serve presentation layer contents & handler business rule layer,these applications are installed in one IBM pSeries RSIC 6000 box ( Configuration : AIX5.1 OS/16GB RAM).Oracle 9i is used as datasource for the entire application,it is installed in another IBM pSeries RSIC 6000 box (Configuration : AIX 5.1 OS/8GB RAM/RAID 5/SAN)
My DayOne experiences
31-Oct
At 23:30hrs,PHI Training facility
Data upload from the old system (It is called as GOLD,which is from Sprient Softwares) to our new system was in progress.(separate team for data migration was working hard)I was verifying the checklists again & again.Everything was perfect,checked up again & again becoz the expected number of users who are going to login to the system was around 800+.
I was walking here & there doing nothing,my PL watched this & cross-checked everything,then said ‘you might hv lots of work tomorrow,so go back to apartments & take some rest’.
01-Nov
At 06.30hrs,PHI Training facility
Data uploads was getting over,I came to office at 0630 & then I cud see everyone,was so tired.Finance peoples was verifying the balances from the old system to our new system.So finally 0800hrs was planned for GoLive with Ramco solutions.
0700hrs I shutdown the application for redeploying the reports & other system admin tasks.
0800hrs I got a call from the finance people & my PL to start the application for production (a welcome mail was sent detailing the application will be online from 0830-0900)
0815 phew…phew..JReports server was not starting,I guess the Jreports application was not shutdown properly,hence it gave java.net.bindexception (port was occupied).
0830 Hence Redeployed/Redirected all the reports to the other testing server & I restarted the webserver,it took around 25mts to reload all the web applications.
0910 Ramco aviation application is up with the Welcome banner at login page.Then it took sometime to check & reload some of the application reports.Got so many calls from users,whether the application is up.My colleagues helped a lot,everyone was patient & they tested the applications to check everything was ok.
0930 After testing all the things,I sent final msg that application is up.
0945 Received calls that Login is too slow & dizzzzzy & all the page launch was too much slow.
1000 Launched weblogic admin console to check whats happening,oh..oh..all the threads stuck & occupied by reports & more over all the memory was consumed by jreports(900mb was set as heap for webserver).I have attached the screenshot of it,checkout at the end of this post.420+users was in the system.
1030 Received calls that page launch was too much slow & some of the field bases cudnt able to launch the production website.
1230 Same condition prevails
1500 To isolate the problem,I ReInstalled Jreports server in the production server again & instead of the jreports running from within the web server,I made it to run as a stand-alone server.But to take effect,I have to restart the server.
02-Nov
0430 Early morning,0430-0600 was the time given by PHI for system maintenance.I redeployed all the reports in the new installation & isolated the Jreports server from the webserver.
0545 Application is up,users started using the system.
0730 Went to Office,users said system performance is good.
0930 Checked up field bases,there too performance was ok.
From there on,system performance scaled up well.Issue is due to reports server,It was sucking the memory & due to which webserver was not getting the memory.And gc was happening at shorter intervals,you cud see the same from the memory graph (this shows the amount of free memory in java heap)
And this was a memorable event of my life..
To learn more about PHI visit www.phihelico.com
SKarthik.info 