What is Security Clearance in the UK?

What is Security Clearance?
National Security Vetting is carried out so that people can work or carry out tasks, which in the course of doing so, requires national security clearance. Government organisations including the Ministry of Defence, Central Government, Defence Estates and the Armed Forces require Security Cleared personnel as well as Companies in the private sector contracted to undertake work for these bodies. Security Clearance levels vary depending upon the sensitivity of the information that is accessed.
The main Security Clearing bodies are:
The Defence Vetting Agency (DVA)
Foreign and Commonwealth Office (FCO)
Metropolitan Police Service (MPS)
How to obtain Security Clearance
You cannot apply for Security Clearance as an individual. Clearance is requested by an employer and carried out by Government agencies. Security Clearance is granted for a specific period of time depending on the employment term or for a particular project.
Security Clearance can be verified and transferred to a new employer if required. If you do not have the Security Clearance required for a particular role you would not be able to start your employment until clearance has been obtained.
You do not have to be a British National in order to gain Security Clearance, but you will have to meet the following criteria depending on the level of clearance required.
There are four main types of national security vetting and clearances:
Developed Vetting (DV) This is the highest level of Security Clearance and is required for people with substantial unsupervised access to TOP SECRET assets, or for working in the intelligence or security agencies. This level of clearance involves Security Check (SC) and, in addition, completion of a (DV) questionnaire, financial checks, checking of references and a detailed interview with a vetting officer. To gain (DV) clearance you will normally have had to have been a resident in the UK for a minimum of 10 years.
Security Check (SC) is for people who have substantial access to SECRET, or occasional access to TOP SECRET assets and information. This level of clearance involves a (BPSS) check plus UK criminal and security checks and a credit check. To gain (SC) clearance you will normally have had to have been a resident in the UK for a minimum of 5 years.
Counter Terrorist Check (CTC) is required for personnel whose work involves close proximity to public figures, gives access to information or material vulnerable to terrorist attack or involves unrestricted access to certain government or commercial establishments. A (CTC) does not allow access, or knowledge, or custody, of protectively marked assets and information. The check includes a Baseline Personnel Security Standard Check (BPSS) and also a check against national security records. To gain (CTC) clearance you will normally have had to have been a resident in the UK for a minimum of 3 years.

Baseline Personnel Security Standard (BPSS) (formally Basic Check)and Enhanced Baseline Standard (EBS) (formerly Enhanced Basic Check or Basic Check +): These are not formal security clearances; they are a package of pre-employment checks that represent good recruitment and employment practice.
A BPSS or EBS aims to provide an appropriate level of assurance as to the trustworthiness, integrity, and probable reliability of prospective employees. The check is carried out by screening identity documents and references.
Other Security checks and clearances:
NATO has four levels of security classification; NATO RESTRICTED (NR), NATO CONFIDENTIAL (NC), NATO SECRET (NS) and COSMIC TOP SECRET (CTS) UMBRA.
NATO's clearance levels function independent of any clearance levels for other nations. However, it is understood that for most NATO nations, granting of a NATO security clearance is handled in a similar manner to that of obtaining a national security clearance.
MPS Vetted
Metropolitan Police Vetting is carried out for all members of the Metropolitan Police Service (police officers, police staff and members of the specials constabulary) Non Police Personnel including Contactors, Contractors representatives, consultants, volunteers and any person who requires unescorted access to MPS premises or uncontrolled access to police information.
The MPS has the following Force Vetting levels:
• Initial Vetting Clearance (IVC)
• Management Vetting (MV)
SIA
The Security Industry Authority operates the compulsory licensing of individuals working in specific sectors of the private security industry within the UK.
The activities licensed under the Private Security Industry 2001 are:
Manned guarding, which includes:
Cash and Valuables in Transit
Close Protection
Door Supervision
Public Space Surveillance (CCTV)
Security guard
Immobilisation, restriction and removal of vehicles
Key Holding
Criminal Records Bureau (CRB) clearance is required for posts that involve working with children or vulnerable adults. Standard Disclosures may also be issued for people entering certain professions, such as members of the legal and accountancy professions. Standard Disclosures contain the following; details of all convictions, cautions, reprimands and warnings held on the Police National Computer (PNC);
Enhanced CRB checks are required for posts involving a far greater degree of contact with children or vulnerable adults involving regular caring for, supervising, training or being in sole charge of such people i.e. Teacher, Scout or Guide leader. Enhanced Disclosures contain the same information as the Standard Disclosures but with the addition of local police force information considered relevant by Chief Police Officer(s).

There are three official criminal record disclosure services within the UK:
CRB provides a service for England & Wales.
Disclosure Scotland is a service provided to manage and operate the Disclosure service in Scotland. Disclosures give details of an individual’s criminal convictions (and in the case of Enhanced Disclosures, where appropriate, non-conviction information).
AccessNI provides a service for Northern Ireland with Disclosures at Basic, Standard and Enhanced levels


Q & A

What is Security Clearance?
Personnel Security vetting is carried out so that people may take certain jobs or carry out tasks that need a national security clearance. These jobs and tasks are located throughout the Ministry of Defence and Armed Forces, as well as in the private sector dealing with defence related work. In addition, a number of other government departments and organisations require Security Clearance.
How do I get a Security Clearance?
First you need a sponsor. Individuals and companies cannot ask for a security clearance unless they are sponsored, and you will not be sponsored unless they are contracted (or are in the process of being contracted) to work on one or more specific MOD classified projects.
For large contracts, an officer in the Defence Procurement Agency (DPA) or Defence Logistics Organisation (DLO) - typically a Project Officer will be your sponsor. For staff in sub-contracted organisations, sponsorship will be provided through the prime contractor.
Why does MOD insist on having sponsors for security clearances? Why can't I just apply for a security clearance?
A security clearance provides a certain level of assurance at a point in time, as to an individual's suitability to have trusted access to sensitive information.
It does not provide a guarantee of future reliability, and all security clearances are kept under review to ensure that the necessary level of assurance is maintained. This review is carried out by Government Departments and Government-sponsored contractors, who are responsible for the oversight and aftercare of individuals, granted a security clearance.
The main types of checks and clearances are listed below and are processed by the following Governments agencies:
Defence Vetting Agency (DVA)
Foreign and Commonwealth Office (FCO)
Metropolitan Police Service (MPS)
National Security Vetting
Developed Vetting (DV) or (DV Cleared) is required for people with substantial unsupervised access to TOP SECRET assets. The following security vetting stages are mandatory before a DV clearance can be approved:
BaselinePersonnel Security Standard (Which is normally undertaken as part of the recruiting process)
Departmental / Company Records Check
Security Questionnaire
Criminal Record Check
Credit Reference Check and review of personal finances
Security Service Check
Check of medical and psychological information provided
Subject Interview and further enquiries, which will include interviews with character referees and current and previous supervisors
On completion of the vetting process, the information collected is assessed and a decision made to refuse or approve a DV clearance.
Once a clearance is granted, it is only valid for a pre-determined period after which a review must be conducted if the clearance is still required. The time interval before a review is required is specified in guidance issued by the cabinet office.
A small number of clearances are granted in spite of some reservations. Risk management requires follow-up work and monitoring of some cases. This activity is termed "aftercare", and may be required in connection with any of the above clearances.
Security Check (SC) or (SC Cleared) is required for people who have substantial access to SECRET or occasional controlled access to TOP SECRET assets. The following security vetting stages comprise a full SC clearance:
Baseline Personnel Security Standard (Which is normally undertaken as part of the recruiting process)
Departmental / Company Records Check
Security Questionnaire
Criminal Record Check
Credit Reference Check
Security Service Check
On completion of the vetting process, the information collected is assessed and a decision made to refuse or approve a SC clearance.
Counter Terrorist Check (CTC) or (CTC Cleared) is required for people who work in close proximity to public figures, or who have access to information or material vulnerable to terrorist attack, or involves unrestricted access to government or commercial establishments assessed to be at risk from terrorist attack. A CTC does not allow access to, or knowledge or custody of, protectively marked assets, but the baseline Personnel Security Standard which is carried out on all MOD personnel and contractors, allows a degree of access. The following security vetting stages are mandatory before a CTC clearance can be approved:
Baseline Personnel Security Standard (Which is normally undertaken as part of the recruiting process)
Departmental / Company Records Check
Security Questionnaire
Criminal Record Check
Security Service Check
On completion of the vetting process, the information collected is assessed and a decision made to refuse or approve a CTC clearance.
What are Employment Checks?
Baseline Personnel Security Standard (BPSS) (formally Basic Check) and Enhanced Baseline Standard (EBS) (formerly Enhanced Basic Check or Basic Check +): These are not formal security clearances; they are a package of pre-employment checks that represent good recruitment and employment practice. A BPSS or EBS aims to provide an appropriate level of assurance as to the trustworthiness, integrity, and probable reliability of prospective employees and should be applied to:
All successful applicants for employment in the public sector and Armed Forces (both permanent and temporary)
All private sector employees working on government contracts (e.g. contractors and consultants), who require access to, or knowledge of, government assets protectively marked up to and including CONFIDENTIAL.
BPSS and EBS are normally conducted by the recruitment authorities or companies to the agreed standard, and because they underpin the national security vetting process it is vital that they are carried out properly and thoroughly and before any further vetting is completed.
Employment Assurance (disclosures) (EA (D)) are required by people from MOD sponsored units and organisations that benefit the MOD, who are being considered for employment with children or vulnerable adults. DVA acts as a co-ordinator for these requests.

Why is the National Security Vetting (NSV)
System necessary and what does it aim to achieve?
The UK needs a security system to protect against threats from hostile intelligence services, terrorists and other pressure groups. Vetting ensures that anyone who goes through the process can be trusted with sensitive government information or property.
Who is affected?
The system applies to people in the following categories whose employment involves access to sensitive Government assets, Crown servants, members of the security and intelligence agencies; members of the armed forces; the police; employees of certain other non-government organisations that are obliged to comply with the Government’s security procedures; employees of contractors providing goods and services to the Government.
How does the vetting system work?
Candidates for jobs that provide access to sensitive information or sites are asked to complete one or more security questionnaires, which invite them to provide the personal details needed to enable the necessary checks to be carried out. Interviews may also be carried out. The depth of checks varies according to the level of regular access to sensitive information that the job entails.
How confidential is the vetting process?
All personal information gathered during the vetting process is handled in the strictest of confidence by the vetting agencies. These bodies include The Defence Vetting Agency (DVA), The Foreign and Commonwealth Office (FCO) and the Metropolitan Police Service (MPS). In a very small number of cases, where serious risks have been identified, a case may be discussed with the Ministry of Defence, security and personnel authorities. In an even smaller number of cases, and only where the person being vetted agrees, line management may be given some relevant information and be asked to help manage the risk. There is an extremely remote possibility of disclosure of vetting information in connection with criminal or civil proceedings.
How do I get a security clearance?
Individuals and companies cannot ask for a Security Clearance unless they are sponsored, and you will not be sponsored unless they are contracted (or are in the process of being contracted) to work on one or more specific MOD / Government classified projects. For large contracts, an officer in the Defence Procurement Agency (DPA) or Defence Logistics Organisation (DLO) - typically a Project Officer will be your sponsor. For staff in sub-contracted organisations, sponsorship will be provided through the prime contractor.
Why can't I just apply for a security clearance?
Security Clearance provides a certain level of assurance, at a point in time, as to an individual’s suitability to have trusted access to sensitive information. It does not provide a guarantee of future reliability, and all security clearances are kept under review to ensure that the necessary level of assurance is maintained. This review is carried out by Government Departments and Government-sponsored contractors, who are responsible for the oversight and aftercare of individuals, granted a security clearance. This would not be possible in the case of private individuals. Security Vetting / Security Clearance is carried out to the following levels by approved government bodies.
Security Vetting / Security Clearance is carried out to the following levels by approved government bodies.
Levels of UK Clearance:
DV / Developed Vetting (MOD)
SC / Security Check
CTC / Counter Terrorist Check
EBS / Enhanced Baseline Standard
BPSS / Basic Personnel Security Standard
NATO / NATO Cleared
MPS / Metropolitan Police Service
SIA / Security Industry Authority
ECRB / Enhanced Criminal Records Bureau
CRB / Criminal Records Bureau

*** I am SC Cleared and working on such a program **

Avatar - The computing and data centre behind making of

The computing and data centre behind making of Avatar

A palm-swept suburb of Wellington, New Zealand is not the first place you'd look for one of the most powerful purpose-built data centers in the world. Yet Miramar, pop. 8,334, is home to precisely that, along with a huge campus of studios, production facilities and soundstages.
The compound is a project that began 15 years ago, inspired by filmmakers Peter Jackson, Richard Taylor and Jamie Selkirk. The studios have since been the main location for creating The Lord of the Rings movies, King Kong, and several others.

Right in the middle sits Weta Digital, the increasingly famous visual effects production house behind high-end commercials and blockbuster movies, most lately the $230 million James Cameron extravaganza AVATAR.
Despite the locale, Weta has drawn plenty of attention. Five Academy Award nominations and four Oscars will do that, but publicist Judy Alley says nothing has matched the buzz of AVATAR. “We’ve done more than 100 interviews in the last few months,” Alley says. With most of the attention focused on the movie’s immersive look, Alley was glad someone was interested to look at the technology installation that sits within Weta and kindly connected us to two of the people who make it run.
As they explained, what makes Weta and a project like AVATAR work is in equal parts the computing power of the data center that creates the visual effects, and the data management of artistic processes that drive the computing.
Hot Gear
Weta Digital is really a visual effects job shop that manages thousands of work orders of intense amounts of data. That preselects most of the fast, constant capacity equipment required. The data center used to process the effects for AVATAR is Weta’s 10,000 square foot facility, rebuilt and stocked with HP BL2x220c blades in the summer of 2008.
The computing core - 34 racks, each with four chassis of 32 machines each - adds up to some 40,000 processors and 104 terabytes of RAM. The blades read and write against 3 petabytes of fast fiber channel disk network area storage from BluArc and NetApp.
All the gear sits tightly packed and connected by multiple 10-gigabit network links. “We need to stack the gear closely to get the bandwidth we need for our visual effects, and, because the data flows are so great, the storage has to be local,” says Paul Gunn, Weta’s data center systems administrator.
That ruled out colocation or cloud infrastructure, leaving Gunn as a sort of owner-operator responsible for keeping the gear running. It also required some extra engineering for the hardware because the industry standard of raised floors and forced-air cooling could not keep up with the constant heat coming off the machines churning out a project like AVATAR.
Heat exchange for an installation like Weta’s has to be enclosed, water cooled racks where the hot air is sucked into a radiator and cycled back through the front of the machines. “Plus,” Gunn says, “we run the machines a bit warm, which modern gear doesn’t mind, and the room itself is fairly cool.”

With building costs absorbed, water cooling becomes much less expensive than air conditioning, and the engineering in the data center allows fine tuning. “I don’t want to give you an exact figure,” says Gunn, “but we’re talking tens of thousands of dollars saved by changing the temperature by a degree.”
Because of passive heat exchangers and the local climate, Weta pays no more than the cost of running water pumps to get rid of heat for all but a couple months a year. Just weeks ago, Weta won an energy excellence award for building a smaller footprint that came with 40 percent lower cooling costs for a data center of its type.
Throughput Revisited
The other half of Weta Digital’s processing story comes from the intense visual effect activities that heat up the data center.
Weta Digital is actually two companies, Weta Workshop, where a crew of artists and craftsmen create physical models, and the like-named Weta Digital, which creates digital effects for commercials, short films and blockbuster movies.
"If it's something that you can hold in your hand, it comes from Weta Workshop," says Gunn, "whereas if it's something that doesn't exist, we'll make it."
In the visual effects process, a mix of inputs come from storyboards, director revisions and tweaking by internal and external digital artists who turn a director’s concept into an image via 3D software from Maya or Pixar’s RenderMan. Artists work through concepts, and iterate versions to get movement and lighting just right. It’s nothing the movie industry hasn’t done all along, says Gunn, only now the tools are different and more data intensive.
The main activity in a visual effects data center is called rendering, the process of turning the digital description of an image into an actual image that can be saved to disk and eventually written to film or another media.
The banks of computers are called render walls, where Joe Wilkie serves as Weta’s “manager wrangler,” the person who oversees the data flow and feeds jobs through the pipeline.
“Wrangler” is a traditional but still common film industry term that first referred to the people who herded the horses and other livestock in Western movies. Likewise, Wilkie says he’s most often called a “render wrangler,” in this case someone who rounds up digital files rather than cattle. “Each part of a movie is an individual item, and it all has to be put together,” he says. “So when an artist is working on a shot, they will hit a button that launches a job on the render wall and loads it into our queueing system.”
The queueing system is a Pixar product called Alfred, which creates a hierarchical job structure or tree of multiple tasks that have to run in a certain order. In any single job, there might be thousands of interdependent tasks. As soon as CPUs on the render wall are freed up, new tasks are fired at idle processors.
At the peak of AVATAR, Wilkie was wrangling more than 10,000 jobs and an estimated 1.3 to 1.4 million tasks per day. Each frame of the 24 frame-per-second movie saw multiple iterations of back and forth between directors and artists and took multiple hours to render.

For Gunn’s data center, that added up to processing seven or eight gigabytes of data per second, a job that ran 24 hours a day for the last month or more of production. It’s a Goldilocks task of keeping the gear running fast, “not too fast, not too slow, but just right,” Gunn says, to keep the production on schedule. “It’s a complex system and when you’re on deadline with a project like this, you really want to make sure the lights stay on.”
A final film copy of AVATAR is more humble than all the back and forth that occurred in its creation: at 12 megabytes per frame, each second stamped onto celluloid amounts to 288 megabytes or 17.28 gigabytes per minute. Deduct the credits from the 166-minute movie and you understand better what the final file consists of.
But the immersive effect of AVATAR comes from the many hours or days of attention to each of about 240,000 frames that go into the final product. Weta asked us to mention vfx supervisor Joe Letteri, who oversaw the interactions of directors, a half-dozen lead concept artists and the supporting artists who made the technology process so intensive and powerful.

OLS with BO Security Model and LDAP in a Data Warehouse

The following are the Business Objects recommendations when creating a security model (see Business Objects Enterprise Administrator’s Guide). This security model should follow these recommendations.
• Use access levels wherever possible. These predefined sets of rights simplify administration by grouping together rights associated with common user needs.
• Set rights and access levels on top-level folders. Enabling inheritance will allow these rights to be passed down through the system with minimal administrative intervention.
• Avoid breaking inheritance whenever possible. By doing so, you can reduce the amount of time it takes to secure the content that you have added to BusinessObjects Enterprise.
• Set appropriate rights for users and groups at the folder level, then publish objects to that folder. By default, users or groups who have rights to a folder will inherit the same rights for any object that you subsequently publish to that folder.
This recommendation cannot be followed due to the access requirements of the reports in the RBAC matrix. In general rights are applied to objects (reports) within the folders.
• Organize users into user groups, assign access levels and rights to the entire group, and assign access levels and rights to specific members when necessary.
• Create individual Administrator accounts for each administrator in the system and add them to the Administrators group to improve accountability for system changes.

What I have found in this installation of Business Objects + Oracle 10g + Oracle Streams + Oracle Label Security is that the OLS labels are not really doing anything additional than what the above BO security model implementation along with LDAP is doing.

I want to discuss and hear more successful implementations of OLS with LDAP and BO Security Models. Any case studies?

A check list for NRIS buying a house in India

A check list for NRIS buying a house in India

Investment from any source in the housing sector is an appreciated aspect in today's indian real estate scenario, from an industry perspective. Let it be Resident Indians, NRIs or even companies, constructing houses creates jobs for a lot of people. A back of the napkin calculation shows that for a 1000 sq feet house, 100 direct employment (architect, building engineer, masons, helpers, electricians, plumbers, painters, carpenters, etc) and over 1000 indirect employment (people working in cement plant, brick kilns, tiles kilns, electrical fittings companies, saw mills, steel plants, paint companies, etc) opportunities are created. Of course the duration of the employment will depend on a number of factors like nearness to a supply sources for material and labour, access to high tech equipment, architecture, etc.

But to construct a house is not all that easy. It is not without substance that a Tamil saying goes, "Veetai Katti Par, Kalyanathai Panni Par" (Basically the saying rates constructing a house and having a child's marriage done among the toughest).

To an already difficult task, the sheer distance and absence during construction become problem multipliers for the NRIs. There was an NRI based out of the USA who got a wonderful sales pitch from a builder. The salesman met the NRI at his office in the USA and arranged for all the documentation and also sent video clippings of the apartment at Bangalore. Convinced on the genuineness, the NRI transferred Rs.50 lakhs to the builder's account. The date for the house warming was fixed after one month. The NRI could not make it to the function due to a pressing office work and had asked his parents to do the poojas.

The parents got the shock of their life, when they landed at the apartment complex the day before the poojas. The complex had only one sample apartment finished (the one in the video). They were told by the Project Manager at the site that the poojas can be done at anytime but the apartment can be delivered only after "6 months".

Another NRI who was building the house himself using an experienced and well referenced engineer found that his house orientation has been shifted by 15 feet. This left him space on the wrong side of the house squashing his plans to build a small commercial complex in future. They now have space for parking 4 cars but none for building a rent worthy space !!!

A number of checks could have been used to be on the safer side in both the above cases:

Thankfully there are a number of professional builders who are a lot more trust worthy. So doing a bit of research on the track record of a builder can help.

For any real estate purchase it is preferable to make visits to the sites before buying them. This exercise is worth it not only because we are committing a large amount of money but also because reversing the decision proves costly as well. If the NRI is not able to make it, he can request a trusted friend or relative to opt for the site visit.

Going for a housing loan through a bank will ensure that the money is released in stages only. This keeps the money safe during the construction. Also, all the banks at their local branches, have their list of shortlisted builders for whose constructions loans are pre-approved. It is better to buy only these constructions, as the banks are quite stringent in their norms for pre-approval and shortlist only those builders who have a proven track record and those project, which comply to all legal norms.

Post the construction, the management of the asset is one of the major issues faced by NRIs. There is no easy solution for this. There are some society associations which support the owners of the buildings with services like maintenance and rent collection. There are again the "friendly neighbor hood real estate agents" who may some times double up as the maintenance manager too. Many times though the "friendly" turn into "greedy" after some time. There are a few professional real estate management firms in most metros, who are now expanding into the 2nd Tier cities too.

Whether the construction rate quoted is for Built-up area or Carpet area? Construction is generally quoted for built-up area and rental is quoted only for the carpet area. There can be a difference of 15 % to 20% between the two based on the type of construction. Today in apartments there is the concept of super built-up area which apart from the built-up area includes stair case, common passages, fire escape passage, etc. The super built-up area can be bloated by as much as 50% of the carpet area.

Robert Allen, the Real Estate Mogul suggests the 100 - 20 - 10 - 1 rule for any real estate purchase. The idea is to check out 100 properties in person; shortlist 20 of them for a deeper scrutiny; enter into negotiation with sellers for 10 of the properties and finally buy the ONE that is best suited.

Technically there should be a check for all the statutory approvals - town planning (Nobody wants a flyover at arms length from the balcony!), water supply and sewage disposal, safety approval from the local fire department, etc. It is always better to ask for the encumbrance certificate and the title deed from the builder to get a legal opinion from a lawyer.

Don't hesitate to ask. This is probably the most important point. Many times, for avoiding being thought of as less intelligent, we question less. For any investing and particularly for real estate the more the questions asked the better the investment. The genuineness of the promoter can be gauged by the patience, the promptness and depth of the answers. Answers like, "Don't worry about that, we will manage", without going into the specifics are danger signs.

Take time. Do not restrain yourself by limiting the time for checking the properties and decision making to the time that you are present in India. A 2 to 4 week holiday cannot be hoped to be converted into a real estate investment period. Start the process before you come here. In case you cannot decide before you leave, it is OK. A Power of Attorney to a parent or a relative can be used to decide on the actual purchase even after you leave the shores of India.
Source: ET

Virga

Did you know that rain can evaporate before it hits the ground?

In meteorology, virga is an observable streak or shaft of precipitation that falls from a cloud but evaporates before reaching the ground.

Virga can cause varying weather effects, because as rain is changed from liquid to vapor form, it removes heat from the air due to the high heat of vaporization of water. In some instances, these pockets of colder air can descend rapidly, creating a dry microburst which can be extremely hazardous to aviation. Conversely, precipitation evaporating at high altitude can compressionally heat as it falls, and result in a gusty downburst which may substantially and rapidly warm the surface temperature.

Virga can produce dramatic and beautiful scenes, especially during a red sunset.

The word virga is derived from Latin, twig or branch. A backronym sometimes found in amateur discussions of meteorology is "Variable Intensity Rain Gradient Aloft."

http://en.wikipedia.org/wiki/Virga

See pic in http://imgur.com/gUZof.jpg and details in http://www.reddit.com/r/pics/comments/9h5ei/rain_rain_go_away/

Chicken Tikka Masala - with yogurt marinated chicken.

Chicken Tikka Masala - with yogurt marinated chicken.

Ingredients:
        Boneless chicken - 1kg. (You may do with bones if you fancy)
        2 hot green chillies
        1 tea spoon coriander seeds( dhaniya)
        1 tea spoon fenugreek seeds
        1+1 tea spoon Turmeric
        1+1 tea spoon 'red chilli' powder
        2+2 table spoon fine grounded masala (mix of spices) - take any 'chicken tikka masala' pack from Tesco.
        2 table spoon oil  - sunflower or olive as you like. Do not use vegetable oil or any other.
       
        2 tea spoon salt
        500g yogurt.
        2 tomatos.
        4 garlic buds (pieces)
Procedure

        [1]
        Marinate chicken in yogurtby mixing 1 tea spoon turmeric, 1 table spoon find grounded masala and                 1 tea spoon red chilli powder. Mix well so that all chicken pieces are immersed in the 'spicy'
        yogurt.
        Leave for min 30 mins.
        [2]
        * Cut the 2 hot green chilles into small pieces.
        * Cut the 2 tomatos into small pieces.
        * cut garlic into small pieces.
        [3]
        * Take a pan.
        * Add 2 tablespoon oil. let it heat a bit may be 2 mins on full gas. (oh yes and switch on gas before this)
        * Add 1 tea spoon fenugreek seeds and 1 tea spoon coriander seeds          
        * Within 4-5 seconds (if oil is really hot) add cut green chillies and cut garlic.
        * Mix for 10 secs or so (again if oil is hot)
        * Add 1 tea spoon turmeric to this and mix for 4-5 secs.
        * Add 2 table spoon fine grounded masala and 1 tea spoon red chillin powder and mix for 5-10 secs.
        * This will not look like a dry mix of spice - lower the gas to medium flame now.
        * Add tomatos and mix really well. Let tomatos kind of melt.
        * Now add the marinated chicken slowly. And mix really well to form a uniform mixture.
        * Increase gas to full and let this boil now for at least 15-20 mins .
                * Occassionally you would need to stir as the mixture may boil and try to come out of pan.
         (You can do this in pressure cooker and you would just need to cook until 3 whistles of cooker)
        * Check now if the chicken is cooked. Use a fork to poke in it and see if it is soft and not like rubber.
        * If chicken is cooked, your curry is ready. Add salt to taste - typically 2 tea spoons for this much mixture.
       
        Eat with rice of bread/nan/chapati etc.


               
       

10 home remedies to avoid swine flu

10 home remedies to avoid swine flu

(Source: Times of India)

Are the rising swine flu casualties giving you jitters? Not sure how you can avoid falling prey to the growing epidemic? First and foremost, there  

is absolutely no need to panic.

Watching television to keep tabs on the progress of H1N1, particularly in the badly affected areas like Pune, is all right. But don't let the hysterical anchors get under your skin and start wearing a mask each time you step out of the house, unless you are visiting a very crowded area. Then too, the mask will protect you only for a specified period.

Without giving in to the swine flu panic and creating a stockpile of Tamiflu and N-95 masks at home and enriching pharma companies, there are a number of other measures you can take to ensure that the virus is not able to get you, irrespective of which part of the world you are in.

It is essential to remember that all kinds of viruses and bacteria can attack you when your immune system is weak, or they can weaken it easily. Hence, building your own defences would be a better, more practical, long-lasting and much more economical idea.

Here are some easy steps you can take to tackle a flu virus of any kind, including swine flu. It is not necessary to follow all the steps at once. You can pick and choose a combination of remedies that suit you best. However, if you are already suffering from flu, these measures can help only up to an extent. And, if you have been infected by H1N1, visiting a hospital and staying in solitary confinement is a must.

1. Have five duly washed leaves of Tulsi (known as Basil in English; medicinal name Ocimum sanctum) everyday in the morning. Tulsi has a large number of therapeutic properties. It keeps throat and lungs clear and helps in infections by way of strengthening your immunity.

2. Giloi (medicinal name Tinospora cordifolia) is a commonly available plant in many areas. Take a one-foot long branch of giloi, add five to six leaves of Tulsi and boil in water for 15-20 minutes or long enough to allow the water to extract its properties. Add black pepper and sendha (salt used during religious fasts), rock or black salt, or Misri (crystalised sugar like lumps to make it sweet) according to taste. Let it cool a bit and drink this kadha (concoction) while still warm. It will work wonders for your immunity. If giloi plant is not available, get processed giloi powder from Hamdard or others, and concoct a similar drink once a day.

3. A small piece of camphor (kapoor) approximately the size of a tablet should be taken once or twice a month. It can be swallowed with water by adults while children can take it along with mashed potatoes or banana because they will find it difficult to have it without any aides. Please remember camphor is not to be taken everyday, but only once each season, or once a month.

4. Those who can take garlic, must have two pods of raw garlic first thing in the morning. To be swallowed daily with lukewarm water. Garlic too strengthens immunity like the earlier measures mentioned.

5. Those not allergic to milk, must take a glass of hot or lukewarm milk every night with a small measure of haldi (turmeric).

6. Aloe vera (gwarpatha) too is a commonly available plant. Its thick and long, cactus-like leaves have an odourless gel. A teaspoon gel taken with water daily can work wonders for not only your skin and joint pains, but also boost immunity.

7. Take homeopathic medicines — Pyrogenium 200 and Inflenzium 200 in particular — five tablets three times a day, or two-three drops three times a day. While these are not specifically targeted at H1N1 either, these work well as preventive against common flu virus.

8. Do Pranayam daily (preferably under guidance if you are already not initiated into it) and go for morning jog/walk regularly to keep your throat and lungs in good condition and body in fine fettle. Even in small measures, it will work wonders for your body's resistance against all such diseases which attack the nose, throat and lungs, besides keeping you fit.

9. Have citrus fruits, particularly Vitamin C rich Amla (Indian gooseberry) juice. Since fresh Amla is not yet available in the market (not for another three to four months), it is not a bad idea to buy packaged Amla juice which is commonly available nowadays.

10. Last but not the least, wash your hands frequently every day with soap and warm water for 15-20 seconds; especially before meals, or each time after touching a surface that you suspect could be contaminated with flu virus such as a door handle or a knob/handle, especially if you have returned from a public place or used public transport. Alcohol-based hand cleaners should be kept handy at all times and used until you can get soap and warm water.

(The author is an avid reader and follower of alternative therapies including spiritual healing, ayurveda, yoga and homeopathy)

How to calculate 95 Percentile of a set of values in oracle?

How to calculate 95 Percentile of a set of values in oracle?

Oracle provides functions to calculate percentile values in a set of ordered data.

Inverse Percentile Functions
Using the CUME_DIST function, you can find the cumulative distribution
(percentile) of a set of values. However, the inverse operation (finding what value
computes to a certain percentile) is neither easy to do nor efficiently computed. To
overcome this difficulty, the PERCENTILE_CONT and PERCENTILE_DISC functions
were introduced. These can be used both as window reporting functions as well as
normal aggregate functions.
These functions need a sort specification and a parameter that takes a percentile
value between 0 and 1. The sort specification is handled by using an ORDER BY
clause with one expression. When used as a normal aggregate function, it returns a
single value for each ordered set.
PERCENTILE_CONT, which is a continuous function computed by interpolation,
and PERCENTILE_DISC, which is a step function that assumes discrete values. Like
other aggregates, PERCENTILE_CONT and PERCENTILE_DISC operate on a group
of rows in a grouped query, but with the following differences:
_ They require a parameter between 0 and 1 (inclusive). A parameter specified
out of this range will result in error. This parameter should be specified as an
expression that evaluates to a constant.
_ They require a sort specification. This sort specification is an ORDER BY clause
with a single expression. Multiple expressions are not allowed.
Normal Aggregate Syntax
[PERCENTILE_CONT | PERCENTILE_DISC]( constant expression )
WITHIN GROUP ( ORDER BY single order by expression
[ASC|DESC] [NULLS FIRST| NULLS LAST])
Inverse Percentile Example Basis
We use the following query to return the 17 rows of data used in the examples of
this section:
SELECT cust_id, cust_credit_limit, CUME_DIST()
OVER (ORDER BY cust_credit_limit) AS CUME_DIST
FROM customers WHERE cust_city='Marshal';
CUST_ID CUST_CREDIT_LIMIT CUME_DIST
Inverse Percentile Functions
SQL for Analysis and Reporting 21-29
---------- ----------------- ----------
28344 1500 .173913043
8962 1500 .173913043
36651 1500 .173913043
32497 1500 .173913043
15192 3000 .347826087
102077 3000 .347826087
102343 3000 .347826087
8270 3000 .347826087
21380 5000 .52173913
13808 5000 .52173913
101784 5000 .52173913
30420 5000 .52173913
10346 7000 .652173913
31112 7000 .652173913
35266 7000 .652173913
3424 9000 .739130435
100977 9000 .739130435
103066 10000 .782608696
35225 11000 .956521739
14459 11000 .956521739
17268 11000 .956521739
100421 11000 .956521739
41496 15000 1
PERCENTILE_DISC(x) is computed by scanning up the CUME_DIST values in each
group till you find the first one greater than or equal to x, where x is the specified
percentile value. For the example query where PERCENTILE_DISC(0.5), the result
is 5,000, as the following illustrates:
SELECT PERCENTILE_DISC(0.5) WITHIN GROUP
(ORDER BY cust_credit_limit) AS perc_disc, PERCENTILE_CONT(0.5) WITHIN GROUP
(ORDER BY cust_credit_limit) AS perc_cont
FROM customers WHERE cust_city='Marshal';
PERC_DISC PERC_CONT
--------- ---------
5000 5000
The result of PERCENTILE_CONT is computed by linear interpolation between rows
after ordering them. To compute PERCENTILE_CONT(x), we first compute the row
number = RN= (1+x*(n-1)), where n is the number of rows in the group and x is the
specified percentile value. The final result of the aggregate function is computed by
Inverse Percentile Functions
21-30 Oracle Database Data Warehousing Guide
linear interpolation between the values from rows at row numbers CRN =
CEIL(RN) and FRN = FLOOR(RN).
The final result will be: PERCENTILE_CONT(X) = if (CRN = FRN = RN), then
(value of expression from row at RN) else (CRN - RN) * (value of expression for row
at FRN) + (RN -FRN) * (value of expression for row at CRN).
Consider the previous example query, where we compute PERCENTILE_
CONT(0.5). Here n is 17. The row number RN = (1 + 0.5*(n-1))= 9 for both groups.
Putting this into the formula, (FRN=CRN=9), we return the value from row 9 as the
result.
Another example is, if you want to compute PERCENTILE_CONT(0.66). The
computed row number RN=(1 + 0.66*(n-1))= (1 + 0.66*16)= 11.67. PERCENTILE_
CONT(0.66) = (12-11.67)*(value of row 11)+(11.67-11)*(value of row 12). These results
are:
SELECT PERCENTILE_DISC(0.66) WITHIN GROUP
(ORDER BY cust_credit_limit) AS perc_disc, PERCENTILE_CONT(0.66) WITHIN GROUP
(ORDER BY cust_credit_limit) AS perc_cont
FROM customers WHERE cust_city='Marshal';
PERC_DISC PERC_CONT
---------- ----------
9000 8040
Inverse percentile aggregate functions can appear in the HAVING clause of a query
like other existing aggregate functions.
As Reporting Aggregates
You can also use the aggregate functions PERCENTILE_CONT, PERCENTILE_DISC
as reporting aggregate functions. When used as reporting aggregate functions, the
syntax is similar to those of other reporting aggregates.
[PERCENTILE_CONT | PERCENTILE_DISC](constant expression)
WITHIN GROUP ( ORDER BY single order by expression
[ASC|DESC] [NULLS FIRST| NULLS LAST])
OVER ( [PARTITION BY value expression [,...]] )
This query computes the same thing (median credit limit for customers in this result
set, but reports the result for every row in the result set, as shown in the following
output:
SELECT cust_id, cust_credit_limit, PERCENTILE_DISC(0.5) WITHIN GROUP
(ORDER BY cust_credit_limit) OVER () AS perc_disc,
Inverse Percentile Functions
SQL for Analysis and Reporting 21-31
PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY cust_credit_limit)
OVER () AS perc_cont
FROM customers WHERE cust_city='Marshal';
CUST_ID CUST_CREDIT_LIMIT PERC_DISC PERC_CONT
---------- ----------------- ---------- ----------
28344 1500 5000 5000
8962 1500 5000 5000
36651 1500 5000 5000
32497 1500 5000 5000
15192 3000 5000 5000
102077 3000 5000 5000
102343 3000 5000 5000
8270 3000 5000 5000
21380 5000 5000 5000
13808 5000 5000 5000
101784 5000 5000 5000
30420 5000 5000 5000
10346 7000 5000 5000
31112 7000 5000 5000
35266 7000 5000 5000
3424 9000 5000 5000
100977 9000 5000 5000
103066 10000 5000 5000
35225 11000 5000 5000
14459 11000 5000 5000
17268 11000 5000 5000
100421 11000 5000 5000
41496 15000 5000 5000
Inverse Percentile Restrictions
For PERCENTILE_DISC, the expression in the ORDER BY clause can be of any data
type that you can sort (numeric, string, date, and so on). However, the expression in
the ORDER BY clause must be a numeric or datetime type (including intervals)
because linear interpolation is used to evaluate PERCENTILE_CONT. If the
expression is of type DATE, the interpolated result is rounded to the smallest unit
for the type. For a DATE type, the interpolated value will be rounded to the nearest
second, for interval types to the nearest second (INTERVAL DAY TO SECOND) or to
the month(INTERVAL YEAR TO MONTH).
Like other aggregates, the inverse percentile functions ignore NULLs in evaluating
the result. For example, when you want to find the median value in a set, Oracle
Database ignores the NULLs and finds the median among the non-null values. You
Hypothetical Rank and Distribution Functions
21-32 Oracle Database Data Warehousing Guide
can use the NULLS FIRST/NULLS LAST option in the ORDER BY clause, but they
will be ignored as NULLs are ignored.

Reference: Oracle Datawarehousing Guide

BI 2.0: Is it really next generation?

We live in real time, minute by minute. News is no longer delayed by days; it is streamed in real time. We bank online and check our real-time balances. We book flights with real-time visibility of seat availability, and we select the seat we want online in real time. All these transactions generate data - lots of data.

To allow us to adapt our business models to today's real-time world, software applications are now built using event-driven technologies. Data moves around in real time over service-oriented architectures (SOAs), using loosely coupled and highly interoperable services that promote standardized application integration.

Yet business intelligence (BI) today has not changed in concept since the invention of the relational database and the SQL query - until the advent of BI 2.0.

BI 2.0 is a term that encapsulates several important new concepts about the way that we use and exploit information in businesses, organizations and government. The term is also intrinsically linked with real-time and event-driven BI but is really about the application of these technologies to business processes.

At the heart of this architecture are events, specifically XML messages. Ultimately, most modern processes themselves are actioned by events. Consequently, when you think about how to add intelligence to modern processes, the humble SQL query looks far from ideal.

The traditional data warehouse has enabled significant advances in our use of information, but its underlying architectural approach is now being questioned. Its architecture limits our ability to optimize every business process by embedding BI capabilities within. We need to look to event-driven, continuous in-process analytics to replace batch-driven reporting on processes after the fact.

In short, how can we build smarter business processes that give our organizations competitive advantage? How can we build the intelligent business?

The Client/Server Legacy

The BI tools most organizations use today were designed to solve a problem that arose in the early 1990s with the spread of the relational database. As more information was stored in databases, simply extracting it became a chore for IT departments because most users weren't interested in becoming experts in writing SQL queries. Getting the data out of databases truly became an end in itself and drove the rise of BI as we know it. Consequently, BI tools today focus on the presentation of data.

As it turns out, though, extracting data that is hours or days old and publishing it into reports, while useful, doesn't provide clear guidance on what users should do right now to improve business performance. As a result, at many companies, BI users don't even review the reports that are sent to them - they relegate them to reference documents. This is often expressed by users who complain that the information arrives too late to be really useful.

Strikingly, this is the antithesis of the real-time, actionable intelligence that many organizations need to provide the quality of service customers demand. At the most basic, such information is a day late and a dollar short in most industries. In retail, for example, three to four percent of potential revenue is foregone due to items not being adequately stocked all the time. The store manager is sent a stock report, but this arrives the next morning, after the close of business and too late to replenish the shelves.

Faster data warehouse queries or prettier dashboard reports — the focus of BI system improvements until now — clearly do not begin to solve the problem because they do not get to the heart of the architectural issue. It is undeniably the case that by the time data has been entered into the data warehouse and then extracted, it is out of date. This isn't a problem for some applications, but it is terminal for those that must run off real-time or near real-time knowledge.

A common misconception is that real-time data isn't needed because there is no way that operations teams could analyze it. This is applying BI 1.0 thinking; simply delivering more reports faster doesn't solve the problem. What's needed is a way to put relevant insight into the hands of operations staff in time to make a difference to day-to-day operations.

Reports are not the optimal deliverable of BI systems. Reports need analyzing and interpreting before any decisions can be made, and there is evidence that users don't look at them until they already know they have a problem. Rather than reporting on the effectiveness of a process after the fact, BI should be used within the process as a way of routing workflow automatically, based on what a customer is doing. In order to do this, you have to not only capture data in real time, but you need to analyze and interpret it as well.

This is essentially event-driven BI - analyzing up-to-the-minute data in the context of historic information - so that actions can be initiated automatically. The data warehouse isn't good at this. Perhaps it is simply being asked to support functions it was not designed to handle.

BI Services Arrive

Over the past few years, companies have started to present their data warehouses as Web services for use by other applications and processes connected by SOA or middleware such as an enterprise service bus (ESB). A fundamental limitation to this approach is that the data warehouse is the wrong place to look for intelligence about the performance of a current process. Real-time process state data, so relevant to this in-process intelligence, is unlikely to be in the data warehouse anyway.

Even layering a BI dashboard onto the data warehouse is inadequate for many operational tasks because they rely on a user noticing a problem based on out-of-date data. Dashboards aggregate and average. They remove details and context and present only a view of the past. Decisions require detail and need to be made in the present.

It's clear that data warehouses will remain, but their role can be clarified as the system of record, as opposed to the only place that BI is done. Reporting and presentation of historical data will continue to be done here - it was designed for that. Given the challenges associated with trying to move to a real-time data warehouse, however, it is clear that information required to support and indeed drive daily operational decisions must come from a different approach to avoid the latency introduced through the extract, transform, load and query cycle.

The Vision for BI 2.0

If the goal of BI 2.0 is to reduce latency - to cut the time between when an event occurs and when an action is taken - in order to improve business performance, existing BI architectures will struggle.

With BI 2.0, data isn't stored in a database or extracted for analysis; BI 2.0 uses event-stream processing. As the name implies, this approach processes streams of events in memory, either in parallel with actual business processes or as a process step itself.

Typically, this means looking for scenarios of events, such as patterns and combinations of events in succession, which are significant for the business problem at hand. The outputs of these systems are usually real-time metrics and alerts and the initiation of immediate actions in other applications. The effect is that analysis processes are automated and don't rely on human action, but can call for human action where it is required.

BI 2.0 gets data directly from middleware, the natural place to turn for real-time data. Standard middleware can easily create streams of events for analysis, which is performed in memory. When these real-time events are compared to past performance, problems and opportunities can be readily and automatically identified.

Intelligent Processes

In order to make a difference to the bottom line, businesses need to make processes smarter. This means either building outstanding ability into automated processes, or providing operations staff with actionable information and changing the day-to-day standard operating procedure to drive data-driven processes. The solution is to leverage the messaging technologies underpinning transactional systems, business process management and SOA, and event-driven real-time BI technologies. These fit together very naturally; you can think of real-time BI as analysis services in an SOA world.

BI 2.0 needs to work with both well-defined processes and less-defined areas. Many processes can't be modeled and explicitly defined using business process management. In fact, the majority of processes today aren't modeled but rather are less explicitly defined. Business users often can't describe their processes accurately enough, and yet operational processes still need intelligence.

BI 2.0 is driven by this need for intelligent processes and has the following characteristics:

Event driven. Automated processes are driven by events; therefore, it is implicit that in order to create smarter processes, businesses need to be able to analyze and interpret events. This means analyzing data, event by event, either in parallel with the business process or as an implicit process step.

Real time. This is essential in an event-driven world. Without it, it is hard to build in BI capabilities as a process step and nearly impossible to automate actions. By comparison, batch processes are informational - they report on the effectiveness of a process but cannot be part of the process itself unless time is not critical. Any application that involves trading, dynamic pricing, demand sensing, security, risk, fraud, replenishment or any form of interaction with a customer is a time-critical process and requires real-time processing.

Automate analysis. In order to automate day-to-day operational decision-making, organizations need to be able to do more than simply present data on a dashboard or in a report. The challenge is turning real-time data into something actionable. In short, businesses need to be able to automatically interpret data, dynamically, in real time. What this means in practice is the ability to compare each individual event with what would normally be expected based on past or predicted future performance. BI 2.0 products, therefore, must understand what normal looks like at both individual and aggregate levels and be able to compare individual events to this automatically.

Forward looking. Understanding the impact of any given event on an organization needs to be forward looking. For example, questions such as "Will my shipment arrive on time?" and "Is the system going to break today?" require forward-looking interpretations. This capability adds immediate value to operations teams that have a rolling, forward-looking perspective of what their performance is likely to be at the end of the day, week or month.

Process oriented. To be embedded within a process in order to make the process inherently smarter requires that BI 2.0 products be process-oriented. This doesn't mean that the process has been modeled with a business process management tool. Actions can be optimized based on the outcome of a particular process, but the process itself may or may not be explicitly defined.

Scalable. Scalability is naturally a cornerstone of BI 2.0 because it is based on event-driven architectures. This is critical because event streams can be unpredictable and occur in very high volumes. For example, a retailer may want to build a demand-sensing application to track the sales of every top-selling item for every store. The retailer may have 30,000 unique items being sold in 1,000 stores, creating 30 million store/item combinations that need tracking and may be selling 10 million items per day. Dealing with this scale is run of the mill for BI 2.0. In fact, this scalability itself enables new classes of applications that would never have been possible using traditional BI applications.

Real-Time, Event-Driven BI

BI 2.0 represents both a bold new vision and a fundamental shift in the way businesses can use information. It extends the definition of BI beyond the traditional data warehouse and query tools to include dynamic in-process and automated decision-making.

In the past, organizations have been forced to rely on out-of-date information and to attempt to fix problems long after they occur. BI 2.0 changes that because it allows BI capabilities to be built into processes themselves - in short, it lets companies create smarter processes.

When BI steps up to identifying problems and initiating corrective actions, not just presenting data, it has definitely evolved. It is ever closer to providing really useful information that can make a difference to the bottom line. Isn't this what BI was supposed to be all along?

Top 5 Trends in BI in 2009

Top 5 trends in Business Intelligence for 2009

Trend #1: Complex Event

Processing (CEP) comes of age

The first generation of successful enterprise data
warehouses uncovered new insights and led to
innovative ways to improve business. These systems are
optimized for one-time queries on mostly static data
captured in the data warehouse long after the event
that generated it has occurred. The paradigm is longlived
data, short-lived queries.
CEP, a logical follow-on from business activity
monitoring (BAM), enables analysis of data streams
and linking of seemingly unrelated events in a
meaningful way. Instead of storing data and having
the execution of a query as the catalyst for results, a
continuous query system effectively "stores" the queries
and new results are initiated by the arrival of new
data, generating real-time insight and/or triggering
appropriate action. (The new paradigm is long-lived
queries, short-lived data).
CEP has heretofore been conspicuously missing from
the mainstream BI arena, necessitating stovepipe CEP
implementations that are only loosely integrated with
organizations' existing visualization, reporting,
dashboarding, information modeling, metadata, and
other BI infrastructure components. We are seeing
indications of that changing as leading BI vendors
partner with and acquire CEP engine providers, and
as BI users incorporate CEP in growing numbers.


Trend #2: Convergence of structured

and unstructured data

This has been a topic of interest for years, but it is
coming up in more conversations with current and
prospective customers than ever before. In the HP
2009 survey, 60% of respondents indicated that they
have an identified need to analyze unstructured data
as part of their BI systems, with over half of those either
doing the unstructured data analysis today, or
developing the capability.
As retailers strive to be more customer-centric, and
healthcare organizations strive to efficiently improve
and manage patient care, and financial institutions
strive to better detect risk and fraud threats, they will hit
limitations by not including unstructured data in the
ever-increasing analysis.
Using only structured data as a proxy for "what is
happening" and making an inference from that,
without correlating with available unstructured data,
can lead to very wrong decisions. For example, coded
diagnoses targeted for the payer often do not indicate
what's really wrong with a patient. Analysis of cancer
case reimbursements might indicate that more money
should be put into brain cancer research because of its
prevalence. But, if any cancer metastasizes to the
brain, it's often coded as brain cancer because of the
greater likelihood of reimbursement. Patient file notes
would indicate the true diagnosis.
Another common business driver is to mine call center
service logs and e-mail together to better understand
customers, for early problem detection and to discern
actual cause of problems.

Trend #3: The line is blurring

between data warehouse,

operational data store (ODS) and

operational systems

Initially, users expected ERP systems to provide needed
reporting. When these systems couldn't meet
requirements due to backlog and overload, users
turned to the data warehouse. Traditional BI satisfies
most strategic reporting and analysis, but not real-time
operational reporting with its associated needs for
high-volume real-time data updates, high availability
needs and high throughput rate of operational queries.
Operational reporting has high overhead and often
ties up the data warehouse, preventing other analytics
from running.
We are seeing operational reporting as a top business
initiative, and increasing interest in the use of a data
provisioning platform as companies need to extend the
data warehouse to more operational use. The platform
needs to go beyond the capabilities of an ODS,
providing operational reporting, data cleansing,
metadata management and data warehouse staging.
Such a data hub enables agility and new applications,
while preserving and enhancing the existing data
warehouse structure, and does it in a much more
efficient and cost-effective manner than using disparate
independent data marts. A hub that connects to
existing enterprise service buses (ESBs) and allows
architectural flexibility, including federation for remote
data, reflects the changing nature of the business while
allowing centralized control over data quality and
data access privileges.
The result is the ability to do operational BI which
involves embedding and automating analytics in a
process so that a person — or another process — can
act on generated information in real time, making
decisions and taking action in the context of a
business process.

Trend #4: Data integration focus

gaining new momentum

Many BI systems in place today were built for strategic
decisions, the sweet spot of traditional BI. Analysis is
done by a small number of people, over a period of
time, allowing for analysts to manually cleanse and
reconcile data from multiple disparate sources, and to
ensure that business rules are applied appropriately
and consistently. Many organizations would like to
increase their intelligence by giving more employees
access to these analytic tools, and applying them to
operational decisions. But it's more than a matter of
increasing capacity for data volumes and query
throughput, and giving the users simpler tools. The
limitation of first generation BI systems is not simply
their inability to handle large volumes of data and
users, but their lack of data integration rigor, including
data cleansing, MDM, and metadata management.
Operational analysis does not afford the time for
manual oversight to ensure proper quality,
reconciliation and classification of the source data,
which may have to be served to applications or
processes where the decisions are then made.
Organizations intent on leveraging their data and
expanding their analytic capability are recognizing the
value of an underlying infrastructure which provides
well-integrated, high quality data to applications,
processes and people.
According to Gartner, "Contemporary pressures are
leading to an increased investment in data integration
in all industries and geographic regions."6 In addition,
"recent focus on cost control has made data integration
tools a surprising priority as organizations realize the
'people' commitment for implementing and supporting
custom-coded or semi-manual data integration
approaches is no longer reasonable."7 The weak
economy will drive M&A in many industries, resulting in
a further need to integrate disparate data to get a
single view of the business, supporting continued
demand for MDM. And financial regulations are likely
to increase. Transparency needed for regulatory
compliance requires a consistent and complete view of
the data which represents the performance and
operation of the business.
In addition to early implementations, we are now also
seeing the results of more recent data warehouse
modernization and data mart consolidation projects
that were undertaken to cut costs, improve performance
and provide more headroom. Where the approach was
to move existing data structures to a new platform to
meet those immediate goals without addressing the
fundamental data integration issues, organizations are
left with the same unwieldy data structure as before,
preventing them from expanding the use of the data
warehouse to meet additional business needs.
Organizations are realizing the need for an overall
enterprise information management (EIM) strategy in
order to leverage data as a corporate asset, to apply
advanced analytics that will help them achieve a
discipline of fact-based decision making, and eliminate
the wastefulness of different teams using different tools
with little consistency and lots of overlap and
redundancy. They are also seeing that data integration
is a critical component of an overall EIM strategy.
Inconsistent meanings create barriers to reliable
analytics. As the boundaries between application
domains like CRM, ERP and product lifecycle
management continue to erode, there is a growing
need to create an enterprise-wide information strategy
to ensure semantic consistency for all users,
applications and services.

----
A7 Gartner, "Magic Quadrant for Data Integration Tools," by Ted Friedman,
Mark A. Beyer, Andreas Bitterer, 22 September 2008.
----

Trend #5: Analytics moves to the

front office — More sophistication in

the hands of business users

Companies are looking to apply advanced analytics to
ERP, CRM and supply chain management systems in
order to achieve strategic competitive differentiation.
The traditional approach to analytics has been to hire
modelers with PhDs who spend three months
developing a model, up to a few dozen or a few
hundred per year. The modeling runs offline to do
customer segmentation, for example. Capturing this
sophistication in tools that can be used by business
managers enables the development and use of not
hundreds, but thousands of models, with a much
shorter time to market. This approach makes it possible
for someone who doesn't know what a neural network
is, to use one, as mainstream capability.
There is an Internet influence on interfaces as well.
Instead of pulling data from multiple sources and
building an analysis cube, the user will go to a portal
and request data elements. Provisioning will be
automated rather than manual, assuming that a data
integration infrastructure has been put in place.