Читать книгу Intelligent Credit Scoring - Siddiqi Naeem - Страница 10

At a minimum, the following main participants are required.

Scorecard Developer

The scorecard developer is the person who performs the statistical analyses needed to develop scorecards. This person usually has:

● Some business knowledge of the products/tasks for which models are being developed. For example, if someone is responsible for building models for an auto loan product or a mobile phone account, they should be familiar with the car-selling business or the cell phone/telco business. Similarly, a person building scorecards for collections needs to understand the collections process. This is to make sure that they understand the data and can interpret it properly in the context of each subject. This would include knowing which types of variables are generally considered important for each product, how decisions and data collection at source impacts quality, and how the model will be used for decision making.

● An in-depth knowledge of the various databases in the company and the data sets being used. The single most important factor in determining the quality of the model is the quality of the data. When the users understand the quirks in the data, where and how the data was generated, deficiencies, biases, and interpretation of the data, they will be able to conduct intelligent analysis of that data. Otherwise, their analysis will be devoid of context. This task may also be covered by someone other than the scorecard developer – for example, a data scientist playing an advisory role.

● An in-depth understanding of statistical principles, in particular those related to predictive modeling. For example, knowledge of logistic regression, fit statistics, multicollinearity, decision trees, and so on.

● A good understanding of the legal and regulatory requirements of models and of the model development process. This includes documentation requirements, transparency, and any laws that control the usage of certain information. For example, in many countries the use of gender, marital status, race, ethnicity, nationality, and the like are prohibited. They would also need to know requirements expected by internal model validation teams so that minimum standards of model governance are met. Detailed knowledge of this subject is usually with model validation groups.

● Business experience in the implementation and usage of risk models. This is related to the business knowledge of the product. If analysts understand the end use of the model, it enables them to develop the one best suited for that task. The analyst will not develop a model that merely meets statistical acceptance tests.

This person ensures that data is collected according to specifications, that all data quirks are taken into account, and that the scorecard development process is statistically valid.

Data Scientist

The data scientist is the person who helps source and extract the required records and fields of information in order to populate the scorecard development database. This person usually has:

● An in-depth knowledge of the various databases in the company, and the data sets being used.

● Proficiency in the tools and systems to determine and document data lineage, to perform field-specific code mappings to common values and definitions from a variety of internal legacy transaction systems and external data reporters.

● Ability to merge/combine information from disparate sources and perform necessary preprocessing to deal with data issues, such as undefined codes, missing information, or extreme/suspect values.

● Familiarity with file formats and fields of information available from the different credit bureaus, rating agencies, and other third-party data providers.

A good example of the required knowledge for data sourcing and extraction is in mortgage lending, where there can be up to four co-applicants, and records for each must be found and joined into a single complete applicant record with individual and combined characteristics derived. These include characteristics such as combined loan-to-value ratio, combined income, payment to combined income, combined debt-to-income ratio, and payment shock to combined current housing payments. Even in a data warehouse, the co-applicant records may reside in a different table that the primary applicant record and matching logic must be used to associate related records. Typical scorecard developers do not possess this type of in-depth knowledge, especially in the larger, more complex financial institutions.

Product or Portfolio Risk Manager/Credit Scoring Manager

Risk managers are responsible for the management of the company’s portfolio and usage of scorecards. They are usually responsible for creating policies and strategies for approvals, credit limit setting, collections treatment, and pricing. In most companies, this person would be the business owner of the scorecard. This person usually has:

● Subject matter expertise in the development and implementation of risk strategies using scores.

● An in-depth understanding of corporate risk policies and procedures.

● An in-depth understanding of the risk profile of the company’s customers and applicants for products/services.

● A good understanding of the various implementation platforms for risk scoring and strategy implementation in the company.

● Knowledge of legal issues surrounding usage of particular characteristics/processes to adjudicate credit applications.

● Knowledge of credit application processing and customer management processes in the company.

● Knowledge of roll rate models; delinquency trends by product, region, and channel; and reports and the average time to charge-off.

When a modeler is asked to build a model (typically a process initiated by the business area), the first question they should ask the businessperson is “why?” That businessperson is typically the risk manager. The answer to that question determines everything else that is done from that point forward, including deciding the target, variable mix in the model, picking the best model, conditions, appropriate model fit measures, and, of course, the final cutoff for any decisions. This person ensures that business considerations are given sufficient thought in the design and implementation of scorecards. Early on in the process, the risk manager can tap their knowledge of the portfolio risk dynamics and performance experience to help with determining the definition of what constitutes “bad” performance for the population of interest. A good practice is to involve risk managers (or a representative) in each phase of the scorecard development process, and get their approval at the end of each one. Risk managers should be able to use some of their experience to point scorecard developers in a particular direction, or to give special consideration to certain data elements. For example, in cases where data is weak or biased, risk managers may use their experience to adjust weight of evidence (WOE) curves or to force certain variables (weak but logical) into the model. Experienced risk managers are also aware of historical changes in the market, and will be able to adjust expected performance numbers if required. They would also contribute heavily to the development of strategies and to gauging possible impacts of those strategies on customers and the various areas of the organization. Scorecards are developed to help in decision making – and anticipating change is key.

Product Manager(s)

The product manager is responsible for the management of the company’s product(s) from a marketing or customer retention perspective. Their main objectives are usually revenue related, and they would have:

● Subject matter expertise in the development and implementation of product-marketing strategies.

● An in-depth knowledge of the company’s typical client base and target markets, including its best/most valued customers.

● Knowledge of future product development and marketing direction.

Product managers can offer key insights into the client base and assist during segmentation selection, selection of characteristics, and gauging impact of strategies. They also coordinate design of new application forms where new information is to be collected. Segmentation offers the opportunity to assess risk for increasingly specific populations – the involvement of marketing in this effort can ensure that scorecard segmentation is in line with the organization’s intended target markets. This approach produces the best results for the most valued segments and harmonizes marketing and risk directions. In other words, the scorecard is segmented based on the profile that a product is designed for, or the intended target market for that product, rather than based on risk considerations alone.

We will cover the gauging of impact on key customer segments post-cutoff selection in later chapters. This involves, for example, measuring metrics like expected approval rates for high-net-worth and similar premium customers. Marketing staff should be able to provide advice on these segments. The product managers typically do not have a say in the selection of final models or variables.

Operational Manager(s)

The operational manager is responsible for the management of departments such as Collections, Application Processing, Adjudication (when separate from Risk Management), and Claims. Any strategy developed using scorecards, such as changes to cutoff levels, will impact these departments. Operational managers have direct contact with customers, and usually have:

● Subject matter expertise in the implementation and execution of corporate strategies and procedures.

● An in-depth knowledge of customer-related issues.

● Experience in lending money.

Operational managers can alert the scorecard developers on issues such as difficulties in data collection and interpretation by frontline staff, impact on the portfolio of various strategies, and other issues relating to the implementation of scorecards and strategies.

A best practice that is highly recommended is to interview operational staff before starting the modeling project. For example, if analysts are looking to develop a mortgage application scorecard, they should go talk to adjudicators/credit analysts who approve mortgages, or other senior staff who have lending experience. Similarly, talking to collections staff is useful for those developing collections models. I normally ask them some simple questions. For applications models, I typically get about 8 to 10 approved and another 8 to 10 declined applications, and ask the adjudicator to explain to me why they were approved or declined. I often ask collectors if they can identify which debtors are likely to pay before talking to them, and which specific variables they use to decide. Staff from Adjudication, Collections, and Fraud departments can offer experience-based insight into factors that are predictive of negative behavior (i.e., variables that they think are predictive), which helps greatly when selecting characteristics for analysis, and constructing the “business optimal” scorecard. This is particularly useful for analysts who don’t have a lot of banking/lending experience, and for cases where you may be developing scorecards for a new product or a new market. In some cases, I have obtained ideas for some interesting derived variables from talking to adjudicators. In one example, when dealing with “thin” files, an adjudicator used the difference in days between the date of the first trade opened and the first inquiry as a measure of creditworthiness. The idea was that a creditworthy applicant would be able to get a credit product soon after applying, while a bad credit would take a while and several inquiries before getting money. Internationally, I have found a lot of nuances in lending, as well as uniquely local variables, from country to country simply by talking to bankers. In Western countries for example, the variable “Time at Address” is useful for younger people as they tend to live on their own soon after turning 18 or graduating. However, in other cultures where young people tend to live with their parents, often into middle age, a high number for that variable may not be fully indicative of financial stability for young people. Interviews with local bankers have helped me understand the data better and construct scorecards that would be most valuable to the business user.

Another good exercise to better understand the data is to spend some time where the data is created. For example, spending time in bank branches, as well as auto and mobile phone dealers, will help understand if and why certain fields are left blank, whether there is any data manipulation going on, and what the tolerance level is for filling out long application forms (relevant when determining the mix of self-declared versus bureau variables to have in the scorecard). This will help gauge the reliability of the data being studied.

In organizations where manual adjudication is done, or where historically applications have been manually adjudicated, interviewing the adjudicators also helps understand data biases. Manual lending and overriding biases data – understanding the policies and lending guidelines, as well as the personal habits of individual adjudicators – will help understand which variables are biased and how. This is similar to studying an organization’s policy rules to understand how its data is biased; for example, if above 85 percent loan to value (LTV), all decisions are exceptions only, then performance for all accounts with LTV greater than 85 percent will be biased and will appear to be much better than reality.

The objective here is to tap experience and discover insights that may not be obvious from analyzing data alone. This also helps interpret relationships later on and identify biases to be fixed. As mentioned earlier, superior knowledge of data leads to better scorecards – this exercise enables the analyst to apply business/experience to the data. Application scorecards are usually developed on data that may be more than two years old, and collections staff may be able to identify any trends or changes that need to be incorporated into analyses. This exercise also provides an opportunity to test and validate experience within the organization. For example, I have gone back to adjudicators and shown them data to either validate or challenge some of their experience-based lending ideas.

Model Validation/Vetting Staff

Model oversight function has always been an important part of the model development process. Their role has become even more critical with the introduction of banking regulations and model risk management guidelines in most countries. The role of model validation and key responsibilities are detailed in documents such as the Supervisory Letter 11-7 from the Federal Reserve Board (SR11-7)12 and Basel II Working Paper 14.13 Ideally, model validation should have:

● A good understanding of the mathematical and statistical principles employed in scorecard development.

● In-depth knowledge of corporate model validation policies, all relevant regulations, and the expectations of banking regulation agencies.

● Real-life experience in developing risk models and scorecards in financial institutions.

● A good understanding of the banking business.

● A good understanding of the data within the bank.

Model validation staff should have regular checkpoints with the model developers and define clearly what is expected in terms of documentation standards. Any divergence from the expected and other red flags should be identified as early as possible.

The better banks have created an environment where the model development, risk management, and model validation teams work in a collaborative way, each with clearly defined roles and accountabilities. This reduces the chances of delays and “surprises,” as well as deadlocks where no one is willing to make a decision. Banks that have long, dysfunctional scorecard development processes usually have:

● Model developers who work in isolation, employ black box processes, and don’t share their knowledge with others.

● Risk management business staff who refuse to participate or are not invited to participate in the scorecard development process, nor share knowledge of how they use the scorecards and downstream decisions.

● Risk management staff who don’t have even the most basic idea of how scorecards are developed.

● People afraid to make decisions because of vague accountabilities.

● Model validation staff who have never built scorecards themselves. This is a major problem with many banks worldwide. Model validation staff who ask the wrong questions and treat the development process as an academic exercise enable the production of statistically perfect but ultimately useless models.

● Model validation staff with no banking experience.

● Vague model validation processes and policies.

Project Manager

The project manager is responsible for the overall management of the project, including creation of the project plan and timelines, integration of the development and implementation processes, and management of other project resources. The project manager usually has:

● Subject matter expertise in the management of projects.

● An in-depth understanding of the relevant corporate areas involved in the project.

IT/IS Managers

IT managers are responsible for the management of the various software and hardware products used in the company. They sometimes have added responsibilities for corporate data warehouses. They usually have:

● Subject matter expertise in the software and hardware products involved in risk management and risk scoring implementation.

● In-depth knowledge of corporate data, data governance policies, and internal procedures to introduce changes to data processing.

● Knowledge of processing data from external data providers.

IT managers can alert scorecard developers to issues related to data collection and coding – particularly when new data is introduced – and to implementation issues related to the software platforms being used to implement scorecards and manipulate data. They must be notified of changes to maintain timelines for implementation. In particular, where scorecards are being developed using complex transformations or calculations, and they need to be implemented on real-time software, the IT department may be able to advise if these calculations are beyond the capabilities of the software. The same is true for derived bureau variables where the derivations have to be done on credit bureau interfaces or using other software. If scorecards are to be implemented within tight timelines, a good idea is to talk to IT to find out how many can be implemented within timelines. This can then drive segmentation strategy, where the number of scorecards to be developed would be consistent with what can be implemented, rather than a larger number.

Enterprise Risk/Corporate Risk Management Staff (Where Applicable)

Enterprise risk departments are responsible for the management of both financial and operational risks at a corporate level (as opposed to the product level). They are usually also involved in capital allocation and oversight of the risk function. They usually have:

● Subject matter expertise on corporate policies on risk management and risk tolerance levels.

● In-depth knowledge of impacts on capital allocation/hedging, and so forth, of introductions to changes in risk adjudication.

● In-depth knowledge of actuarial practices.

Enterprise risk staff is usually advised when new strategies change the risk profile of the company’s portfolio. Increasing or decreasing risk levels affect the amount of capital a company needs to allocate. Taking significant additional risks may also be in contravention of the company’s stated risk profile target, and may potentially affect its own credit rating. Enterprise risk staff will ensure that all strategies comply with corporate risk guidelines, and that the company is sufficiently capitalized for its risk profile.

Legal Staff/Compliance Manager

Credit granting in most jurisdictions is subject to laws and regulations that determine methods that can be used to assess creditworthiness, credit limits, and characteristics that cannot be used in this effort. A good practice is to submit a list of proposed segmentation and scorecard characteristics to the legal department, to ensure that none of them is in contravention of existing laws and regulations. In the United States, for example, issuing arising from the Equal Credit Opportunity Act,14 Fair Housing Act,15 Dodd-Frank,16 Regulation B,17 as well as “adverse” and “disparate” impact are all areas that need to be considered during scorecard development and usage.