Showing posts with label AI. Show all posts
Showing posts with label AI. Show all posts

Wednesday 13 March 2024

How GenAI models like ChatGPT will end up polluting knowledge base of the world.

 Around nine years ago, around the PI day, I wrote a blog post about an ancient method of remembering digits of pi. Here is the link to blog post.

Here is the operative part of the blog post.

The code is as follows.

क ख ग घ ङ च  छ  ज  झ 
ट ठ ड ढ ण त थ द ध 
प  फ  ब  भ  म  
य  र  ल  व  श  ष  स  ह  क्ष  

With the above key in place, Sri Bharathi Krishna Tirtha in his Vedic Mathematics gives following verse.
गोपी भाग्य मधुव्रात  श्रुङ्गिशो दधिसन्धिग  |
खलजीवित खाताव गलहालारसंधार |
If we replace the code from the above table in the above verse, here is what we get.
31 41 5926 535 89793
23846 264 33832792
That gives us 

Today, I asked ChatGPT this question.

If you look at the details explanation provided by ChatGPT, it is completely wrong. What I picked up is from a book on the topic. Once more and more such content floods internet, it will become extremely hard to decipher what is right and what is wrong. Furthermore even academic text being written in future would consist of ChatGPT created stuff.

Unles GenAI can provide traceability to the source material, I don't think it is worth giving the attention that it is being given.

Wednesday 20 February 2019

The morality of a machine

It is 2019, everybody is gung-ho on about Artificial Intelligence and/or Machine Learning. If we look around, there are many systems that exist today which incorporate AI/ML in some shape or form. Most systems in product consist of typical classifiers or regression that provides information to make a particular decision. The question of the morality of a machine has not found top consideration because till now the AI/ML is producing output that is used by a human to make actual decision.
When Google Photo image tagging algorithm misclassifies an image and tags it with somebody else's’ name, it is not a serious problem because I can personally go ahead and reclassify the photo with the correct name. Still, the algorithm is correctly classifying more than 90% of images accurately and reducing my workload. When Google Mail or Gmail misclassifies one of the important email containing a notice from the income tax department, it is a significant hassle but still something that I can easily fix. The primary point here is that all the input created by AI is brought to the notice of a human who looks at all the available data before making any decision.
But, the day is not far off when we will reach a situation where machines may be directly handling responses and humans are either not in the loop or they become so unmindful of what the machines are doing for you that just ignore those actions. For example, it is not inconceivable to expect Gmail to automatically respond to any notice received from the income tax department by looking at your financial data that is stored on some Google Drive. It does not have the capability today but the technology exists where this capability can be easily built.
The problem becomes more compounded where the actions taken by machines may result in serious hazards to humans. Look at all the excitement around self-driving cars. When human’s are driving cars, they are constantly making decisions that are driven by morality rather than plain logic. Take an example where you are driving your car and on one side there is a pedestrian or a cyclist while on the other side of your car there is another car. Most human would err on the side of the car because that is just the material damage. Our morality expects us to value human life higher than material things. If we extend this problem, let’s say the self-driving car AI realizes that it can’t avoid an accident, it has to choose between a child and one aged person. Suddenly the decision becomes extremely difficult. Humans make these decisions instinctively and many are disturbed through decisions for many years down the line.
Should you value the life of a child higher than a senior citizen? What about when you have to choose between parent and child? What about choosing between the head of state and a normal person? These are complex decisions. We can write code that can make these decisions but the rules have to be defined and agree by society and governments. If every car manufacturer starts making their own moral decisions, that may become anarchy. Governments, Societies, Courts need to define a set of morality rules that every autonomous AI has to follow. I think the time has come when Asimov’s three laws need to be expanded to something like Asimov’s three laws and other morality principles. These principles need to define how a machine can evaluate different outcomes based on a given situation and choose an outcome that is acceptable by courts, societies, and governments. Unless that happens, AI should be relegated to a decision support system and should not become something that has any control.

Saturday 24 November 2018

A pothole detector

I like to drive a lot and that and during highways at high speeds encountering a pothole can be tremendously dangerous. I always wondered whether AI can really detect potholes in the road. So I decided to give it a try. I always have a dashcam on my car that constantly records videos of my drive. So I decided to take a few of the videos along with the pothole dataset available on the internet to train a pothole detector.

I visualized this problem as an object detection problem. As I approached the problem, it became very clear that the form in which the dataset was, it was not ready for training. Look at one of the pictures from the dataset.

The image contains a bunch of potholes. We can use these type of images if we were to build a classifier which classifies the images into two groups, one having potholes and another without potholes.
Our intentions are very different. What we want to achieve is to locate the pothole in the frame. So we have to label our images. That is a time-consuming activity.
There are many tools available that help in tagging and labeling. I looked at quite a few of them and finally found VoTT from Microsoft to be a good tool. The fact that it can export tagged data into multiple formats was an icing on the cake.
So, I got around to tagging a bunch of these images. At this point, the only object that we are interested in is Pothole. After tagging, the above image looked like below image.
We also did the same exercise with a bunch of dashcam videos. It is really a time-consuming exercise and for that reason, we did not really get a substantial dataset.
Once we have our tagged and labeled dataset ready, we need to export them to tfrecord format. This is the format that tensorflow is most familiar with and it also makes it easy to merge multiple datasets into one.
Once, we are done with this, we have a number of tfrecord files and a pbtxt file.
item {
 id: 1
 name: 'RoadBump'
item {
 id: 2
 name: 'Pothole'
item {
 id: 3
 name: 'People'
item {
 id: 4
 name: 'Truck'
item {
 id: 5
 name: 'Bus'
item {
 id: 6
 name: 'Car'
item {
 id: 7
 name: 'TwoWheeler'
item {
 id: 8
 name: 'AutoRickshaw'
item {
 id: 9
 name: 'BadRoad'

Now we create two subdirectories, one to hold training data and another to hold evaluation data. We then split tfrecord files randomly into two sets, one for training and another for evaluation. Both the subdirectories can have the same pbtxt file.
The next step is to create a google cloud project. I am doing this work on an Ubuntu Linux machine. Download Google Cloud SDK to make life easier. The Google Cloud Developer tools website has good information on what needs to be done, but in summary, you need to authorize a user account.
The next step is to install tensorflow. Even though I plan on running training on Google Cloud, still I need the tensorflow because there are utilities that are required to package the job.
In my experience, tensorflow still works better with python 2.7, it works with python 3.5 and 3.6 as well but if you have a choice, stay with python 2.7.
You have a choice of installing tensorflow or tensorflow with GPU. If you have a machine that has Cuda, you need to install Cuda 9.0 drivers. If you are using Ubuntu 18.04, by default Nvidia will install Cuda 10.x. You have to make sure you remove them and install Cuda 9.0 drives. Also when you uninstall Cuda 10.x drivers, be careful. In my case it remove some of the important packages and then I had to manually install them later.
$ sudo apt update
$ sudo apt install python-dev python-pip
$ sudo pip install -U virtualenv  # system-wide install
$ pip install --upgrade tensorflow 
$ pip install --upgrade tensorflow-gpu # GPU install

It may be prudent to configure a virtual environment setup to make sure you don't have to worry too many broken dependencies. Tensorflow website has good information on how to go about doing the installation and that would be the best resource to go about it.
The next step is to figure out what should be done about the model. Since I am modeling the problem as an object detection problem, I will use one of the existing model checkpoints to use transference learning.
$ git clone

We go into models/research directory and the code for object detection is in the object_detection directory. Tensorflow model zoo contains pre-trained models. We decided to use faster_rcnn_inception_resnet_v2_atrous_coco as the base model for training. At this time, download the model and extract the archive in a directory. It is always suggested to pick up the pipeline config file from the git repository. The config files are contained in model/research/object_detection/samples/configs directory. Each model has a corresponding config file. I have used faster_rcnn_inception_resnet_v2_atrous_coco.config for my model.
Now we need to set up our cloud environment. The first step is to create a bucket in cloud storage. Google tutorial uses the following trick to give names to buckets.
$ export PROJECT=$(gcloud config list project --format "value(core.project)")
$ export YOUR_GCS_BUCKET="gs://${PROJECT}-ml"
Within the cloud bucket, we create a directory named data. Within data, we create two subdirectories, train, and eval and we copy corresponding .tfrecord and pbtxt files into those directories. So ${YOUR_GSC_BUCKET}/data/train contains training dataset and ${YOUR_GSC_BUCKET}/data/eval contains evaluation dataset.
We need to package pycocotools to be submitted along with the model.
$ cd model/research
$ bash object_detection/dataset_tools/ /tmp/pycocotools
We also need to add the location of models folder and models/slim folder to PYTHONPATH variable. From models directory, we need to compile protobuf files.
$ protoc object_detection/protos/*.proto --python_out = .

Now we generate all the archives that are needed to run the training. From model directory, we run following commands.
$ python sdist
$ (cd slim && python sdist)
Now we need to configure pipeline_config file. We use faster_rcnn_inception_resnet_v2_atrous_coco.config file as the base file. Basically, we need to modify all the instances of PATH_TO_BE_CONFIGURED in the file with appropriate values. We will need to set the following values.

  1. fine_tune_checkpoint the cloud location of the model checkpt.
  2. input_path for training data. This should be a cloud location of training files. You can give wild cards here. For example gs://myproject-ml/data/train/*.tfrecord.
  3. label_map_path for training data. For example gs://myproject-ml/data/train/tf_label_map.pbtxt
  4. input_path for eval data. This should be a cloud location of eval files. You can give wild cards here. For example gs://myproject-ml/data/eval/*.tfrecord.
  5. label_map_path for eval data. For example gs://myproject-ml/data/eval/tf_label_map.pbtxt

For convenience sake, I also renamed the config file to pipeline.cfg. Now we upload the cfg file to cloud storage in data directory.
Now we are ready to submit our job for training.
$ gcloud ml-engine jobs submit training `whoami`_object_detection_`date +%s`     \
      --packages dist/object_detection-0.1.tar.gz,slim/dist/slim-0.1.tar.gz,/tmp/pycocotools/pycocotools-2.0.tar.gz
      --module-name object_detection.model_main \
      --config train.yaml \
      --runtime-version 1.10 \
      -- \
      --pipeline_config_path=${YOUR_GSC_BUCKET}/data/pipeline.config \
Please pay attention to the fact that last two command line arguments are after -- and are passed on to the model.
In my case, I ran the model for approximately 24 hours and even with insufficient data the results were remarkable. Here are some of the images with detection in action.

As we can see even a limited amount of training data and training compute cycles can produce remarkable results. Now as a next step we need to tag and label a larger amount of data and then train it for longer.

Friday 8 December 2017

Finding representative phrases

As part of my doctoral research, I was faced with an interesting classification problem. I was working with a dataset extracted from the Usenet archive. In Usenet, the content is automatically classified within newsgroups. my challenge was to find representative phrases for a given content based on the primary classification and content of the Usenet post.
Identifying topics
Because we are dealing with technical text, we created our own list of stop words that we ignored while processing for representative phrases. The code appended below is looking of ngram length of maximum 2 but can be easily changed for larger length.

Saturday 1 March 2014

Building a natural language classifier

The idea behind this post is to build a classifier that would work based any content. For the purpose of this experiment, we chose twitter data.
In any machine learning experiment, we need to have a training set and a test set. I collected a number of tweets during the period October 2013 to February 2014 using twitter stream API. Total 322,382 tweets were collected during this time period. Total 235,100 users participated in this information exchange. Based on specific events on days, I used the following four tokens to filter the tweets on stream API that are 1) India 2) FamilyGuy 3) FastAndFurious 4) Thanksgiving. The data that I collected contains all the original tweets initiated during this time and all the retweets. Since Twitter does not maintain the flow of retweets across the users, all the retweets point to the original tweet rather than intermediate tweets. Because of this even if a particular tweet is missed but retweets are seen, we can always find the original tweet from retweet since original tweet is always fully embedded in the retweet.
Twitter schema allows users to add metadata to their tweets. This metadata manifests itself in the form of hashtags in the tweets. Hashtags provide a primary means of categorization. Since tweets are limited in size of content that they can carry, users can attach URLs that point to a larger content. We want to evaluate the impact of presence or absence of hashtags and URLs in a tweet.
We parse the tweet data and create a table with following columns for each tweet.

  1. Retweets in less than 10 seconds 
  2. RTs in greater than 10 and less than 30 seconds 
  3. RTs in greater than 30 and less than 1 minute 
  4. RTs in great than 1 min and less than 5 mins 
  5. RTs in great than 5 mins and less than 10 mins 
  6. RTs in greater than 10 mins and less than 30 mins 
  7. RTs in greater than 30 mins and less than 1 hour 
  8. RTs in greater than 1 hour 
  9. Number of hashtags 
  10. Number of URLs 
  11. Total retweets 
  12. Category of popularity of retweets denoted by an interval variable ranging from 1 to 7. 1 being the least popular while 7 being most popular

Since we have already seen that there is a large percentage of Twitter users who primarily just retweet and don’t add any content of their own. Also, Twitter does not really preserve the path traveled by a tweet. For all the further analysis, we only take into account tweets that were created by a user. We ignored the instance of retweets by other users since it is not adding any information.
The next step is to find appropriate categories for the tweet. We use the following set of rules to find appropriate categories for the tweets being examined.
  1. Look at all the hashtags in a tweet. All of these are potential categories
  2. For tweets having more than one hashtag, we only categorize it under the hashtag with the highest frequency
  3. If a tweet does not have any hashtag, we classify it under the URL which is attached to it.
Once we have collected all the hashtags and URLs, we need to eliminate similarities. While deciding on the categories of the user, it is important to notice that many times people do mistakes in typing hashtags, use different spelling and also use related words to categorize a tweet. 
Considering all these issues, we propose the following algorithm to come up with relevant categories of tweets.
  1. We measure the distance between two hashtags to identify similar hashtags. We use Jaro Winkler distance as a measure of distance between two hashtags. One of the reasons for choosing Jaro Winkler distance is the fact that it is best suited for short strings which is the case with Hashtags. At the end of step 1), we have similar hashtags clubbed into a single bucket. For example, we can see that the above algorithm identifies "familyguyxxx familyg familyguyproblems familyguyfans" hashtags as a similar one.  
  2. At the second step, we look at groups of hashtags and then look at the hashtags that occur together very frequently and then we merge that group. For example, a group containing “familyguy” and a group containing “briangriffin” occurs together frequently and we merge them in a single group
  3. We eliminate all the hashtags that seem to occur together but the text contained in those tweets has very high distance. It is a common practice among the spammers to ride popular and trending hashtags and start tweeting by including that hashtag.
Thus we end up with the final list of categories that we can use in final analysis. These are our significant categories. The next step is to build a classifier. We have used a dynamic language model based classifier. The Dynamic Language Model based classifiers use multivariate estimators for the category distribution and dynamic language models for the per-category character sequence estimators. The whole data is segregated into training and test set by dividing it into 90% and 10%. The training set is 90% and the test set is 10%.
Total accuracy of the model thus built is 83.64%. This means that out of all the testing cases, the model is able to successfully classify 83.64% of times accurately. 

Important results
Total cases
True Positives
False Negatives
False Positives
True Negatives

You can find the complete IEEE Paper by clicking on hyperlink.

Friday 7 October 2005

Using Neural Networks to Explain Behavior of Indian Markets

It is believed that stock prices are impacted by three factors.

  • Company/Industry performance
  • Macroeconomic outlook
  • Sentiment-driven by news
The first two of the above are defined by numerical data. I decided to build an artificial engine that can predict the value of a particular stock based on movement in the above variables. We looked at following variables. 
  • assetsWithBankingSystem – Total assets with the banking system
  • bankCredit – Bank credit in India
  • cash – Cash in hand
  • investmentAtBookValue – Total bank investments at book value
  • liabilitiesToBankingSystem – Total liabilities of banks to the banking system
  • liabilitiesToOthers – Total liability of banks other than the banking system
  • curcredit – Current account credit in INR
  • curdebit – Current account debit in INR
  • capcredit – Capital account credit in INR
  • capdebit – Capital account debit in INR
  • errcredit – Errors credit
  • errdebit – Errors debit
  • balcredit – Balance credit
  • baldebit – Balance debit
  • monmovcredit – Monetary movements credit
  • monmovdebit – Monetary movements debit
  • callMoneyHigh – Call money rate, High
  • callMoneyLow – Call money rate, Low
  • eps – Earning per share of the company
  • ceps – Cash earning per share of the company
  • bookValue – Book value of the company
  • div – Dividend paid per share of the company
  • opProfitPerShare – Operating profit per share of the company
  • netOperatingIncomePerShare – Net operating income per share of the company
  • freeReserves – Free reserves with the company
  • opm – Operating profit margin of the company
  • gpm – Gross profit margin of the company
  • npm – Net profit margin of the company
  • ronw – Return on net worth of the company
  • debtToEquity – Debt to equity ratio of the company
  • currentRatio – Current ratio of the company
  • quickRatio – Quick ratio of the company
  • interestCover – Interest cover of the company
  • salesByTotalAssets – Sales by total assets of the company
  • salesByFixedAssets – Sales by fixed assets of the company
  • salesByCurrentAssets – Sales by current assets of the company
  • noOfDaysOfWorkingCapital – No of days of working capital with the company
  • cpi – Consumer price index
  • br – Bank Rate
  • idbiRate – IDBI minimum term lending rate
  • maxCMR – Maximum Call Money Rate
  • maxPLR – Maximum prime lending rate
  • minPLR – Minimum Prime lending rate
  • price – Crude price
  • totalINRdebt – Total debt in Indian Rupees
  • concessionalDebtAsPercOfTotal – Concessional debt as a percentage of total
  • shortTermDebtAsPercOfTotal – Short-term debt as a percentage of total
  • affConstant – Agriculture, Forestry and Fishing, GDP factor cost, Constant prices
  • affCurrent – Agriculture, Forestry and Fishing, GDP factor cost, Current prices
  • cspsConstant – Community social and personal services, GDP factor cost, Constant prices
  • cspsCurrent – Community social and personal services, GDP factor cost, Current prices
  • consConstant – Construction, GDP factor cost, Constant prices
  • consCurrent – Construction, GDP factor cost, Current prices
  • egwsConstant – Electricity, Gas and Water Services, GDP factor cost, Constant prices
  • egwsCurrent – Electricity, Gas and Water Services, GDP factor cost, Current prices
  • firebsConstant – Finance, Insurance, Real Estate, and Business services, GDP factor cost, Constant prices
  • firebsCurrent – Finance, Insurance, Real Estate, and Business services, GDP factor cost, Current prices
  • manuConstant – Manufacturing, GDP factor cost, Constant prices
  • manuCurrent – Manufacturing, GDP factor cost, Current prices
  • maqConstant – Mining and quarrying, GDP factor cost, Constant prices
  • maqCurrent – Mining and quarrying, GDP factor cost, Current prices
  • tdpConstant – Total domestic product, GDP factor cost, Constant prices
  • tdpCurrent – Total domestic product, GDP factor cost, Current prices
  • thrConstant – Trade, Hotel and Restaurant, GDP factor cost, Constant prices
  • thrCurrent – Trade, Hotel and Restaurant, GDP factor cost, Current prices
  • aff – Agriculture, Forestry and Fishing, GDP factor cost
  • csps – Community social and personal services, GDP factor cost
  • cons – Construction, GDP factor cost
  • egws – Electricity, Gas and Water Services, GDP factor cost
  • firb – Finance, Insurance, Real Estate, and Business services, GDP factor cost
  • manuf – Manufacturing, GDP factor cost
  • min – Mining, GDP factor cost
  • tdp – Total domestic product, GDP factor cost
  • thr – Trade, Hotel and Restaurant, GDP factor cost
  • currencyWithPublic – Total currency with Public
  • m3 – Money supply, also referred to as stock of legal currency in the economy
  • timeDepositsWithBank – Total time deposits with the bank
  • totalIncome – Total income of RBI
  • totalExpenditure – Total expenditure of RBI
  • netAvailableBalance – Net available balance in RBI
  • surplusToCentralGovernment – Surplus payable to central government from RBI
  • totalIssuesLiabilities – Total liabilities, Issues
  • totalIssuesAssets – Total assets, Issues
  • totalBankingLiabilities – Total liabilities, Banking
  • totalBankingAssets – Total assets, Banking
  • reserveMoneyLiabilities – Reserve Money, Liabilities
  • reserveMoneyAssets – Reserve Money, Assets
  • forwardCashSpot – Forward Cash Spot, USD forward premia
  • forwardCashOneMonth – Forward Cash one month, USD forward premia
  • forwardCashThreeMonth – Forward Cash three months, USD forward premia
  • forwardCashSixMonth – Forward Cash six months, USD forward premia
  • forwardCash12Month – Forward cash twelve months, USD forward premia
  • referenceRate – RBI reference rate for USD
  • rate – US interest rate
  • quantitiy – Quantity of particular stock traded
  • turnover – Total turn over of stock traded
We collected the data for the above metrics and established their relationship with the following data specific for a stock.
  • Previous day close
  • Day open
  • Day high
  • Day low
  • Day close
Since there are a very large number of input variables related to economic indicators which may have a heavy correlation between themselves, the factor analysis was used to reduce the features to a  manageable set of features that were used as inputs for the neural network later to develop the prediction model. For each company, four models were constructed as follows.
  • 1D model, which would predictions the prices for next day given the stock price, turnover, and quantity for a day earlier to the previous day.
  • 7D model, which would make predictions  given the stock price, turnover and quantity for a week earlier
  • 15D model, which would make predictions  15 days down the line.
  • 180D model, which would make predictions six months down the line given the
  • stock price for a day.
After the factor analysis of the data, 96 inputs are reduced to 20 inputs with 95% of the variance explained. These factors are as follows. As we go to later factors, these mostly cover the residual values from initial factors.
  • Factor 1 – RBI influence and Core sector
  • Factor 2 – Foreign Exchange and Crude
  • Factor 3 – Agriculture, Total Domestic Product
  • Factor 4 – Company Financials
  • Factor 5 – Company Ratios
  • Factor 6 – Agriculture, Community services, debt structure with RBI
  • Factor 7 – Company Capital structure, profitability ratios, and other indicators
  • Factor 8 – Banking system residuals
  • Factor 9 – Company Liquidity Ratios
  • Factor 10 – Company stock performance
  • Factor 11 – RBI balance sheet debt structure and errors
  • Factor 12 – RBI balance sheet errors
  • Factor 13 – Company indicators (residuals)
  • Factor 14 – Banking system residuals
  • Factor 15 – Company financial ratios, Residuals
  • Factor 16 – Foreign Exchange, Crude and interest rate, Residuals
  • Factor 17 – Company Financial Ratios, Residuals part 1
  • Factor 18 – Company Financial Ratios, Residuals part 2
  • Factor 19 – USD Forward Spot rate
  • Factor 20 – IDBI lending rate and crude prices
The companies in NSE-50 index were considered.

Design of neural networks

Inputs and outputs

The economic indicators for model related to the company have been factored into 20 factors that explain most of these numbers. Additional 3 inputs are company specific and are related to the past stock price data with respect to that company.
  • Previous Close
  • Previous Turn Over
  • Previous Quantity
These makeup for the 23 variables that are used as inputs for the neural network. Three different
neural networks are used for the following three output variables
  • High
  • Low
  • Close

Hidden Layers

It is assumed given the richness of the data that at least 2 hidden layers would be required to form a meaningful neural network. The neural network will have 23 inputs and will have 1 output. Different neural networks would be created and a training run would be performed for 1500 cycles of data set. At the end of the sample run, the best network would be chosen for further training.
Neural networks that were evaluated are with
  • 1 input layer with 23 inputs
  • first hidden layer with nodes 31 to 351
  • second hidden layer with nodes 8 to 31
  • 1 output layer
The neural network with hidden layer 1 of 130 nodes and hidden layer 2 of 17 nodes comes with best error values to be further used. I have used libraries provided by Joone. Following are the main fragments of code for this exercise.


Prediction Six Months


This exercise concludes that there is merit to using neural networks in trying to understand and predict the behavior of markets but with a certain caution. Following are important points to be kept in mind if this model is used for investment decisions.

  • The model does not return profitable results in very short duration trades, the investor should have an investment horizon of more than 6 months for the model to work properly
  • The model does not guarantee that all the trades would be profitable but overall there is a better chance of profits
  • Stocks with less volatility perform better in model-based prediction
For more details please refer to the report attached below.

How GenAI models like ChatGPT will end up polluting knowledge base of the world.

 Around nine years ago, around the PI day, I wrote a blog post about an ancient method of remembering digits of pi. Here is the  link  to bl...