TechVision

**Title: How Language Models Generate Text: A Peek Under the Hood**   Have you ever wondered how AI tools like ChatGPT or Gemini craft coherent sentences, answer questions, or even write code? The secret lies in a process called **autoregressive text generation**—a method that powers most modern neural language models (LMs). Let’s break down how it works!   --- ### **Step 1: Start with a Prefix**   Imagine you type the phrase *“The cat sat on the”* into an AI chatbot. This input is called the **prefix**, and the LM’s job is to predict what comes next.   --- ### **Step 2: Predict the Next Token**   Using its neural network (often a Transformer-based architecture), the LM analyzes the prefix and generates a **probability distribution** over its fixed vocabulary. For example, it might assign:   - 60% probability to *“mat”*   - 30% to *“rug”*   - 10% to *“floor”*   This distribution reflects the model’...

Apache Stanbol - Website  - https://stanbol.apache.org/docs/trunk/tutorial.html Stanbol helps to model a semantic relationship around NLP. Given a document it can find the main concepts like NER and gives link to these entities into DBPedia or Enterprise database. The steps to follow to use Stanbol : 1) Use RESTFul aPI 2) Use Java API Using RestFul API ---------------------------------- Step 1: export MAVEN_OPTS="-Xmx1024M -XX:MaxPermSize=256M" Step 2 : svn co http://svn.apache.org/repos/asf/stanbol/trunk stanbol Step 3:  mvn clean install (From downloaded stanbol directory) Step 4: java -Xmx1g -jar stable/target/org.apache.stanbol.launchers.stable-{snapshot-version}-SNAPSHOT.jar (give your corresponding stanbol jar name) Step 5 : Open http : // localhost : 8080in web browser Step 6 : The stanbol options are available now. For ex. enhancer we can use as we click on that and give a text , we will get the corresponding NERs and its related DBPedia links. Oth...

Top machine learning libraries for Python 1. Numpy Numerical Python It is the most fundamental package for scientific computing in python. It provides operations for matrix and array. Numpy arrays are used in most of the ML projects. The library provides vectorization of mathematical operations on the NumPy array type 2. Scipy modules for linear algebra, optimization, integration, and statistics. It contains modules for linear algebra, optimization, integration, and statistics. 3. Pandas It works with labelled and relational data.  It designed for quick and easy data manipulation, aggregation, and visualization. Here is just a small list of things that you can do with Pandas:     Easily delete and add columns from DataFrame     Convert data structures to DataFrame objects     Handle missing data, represents as NaNs     Powerful grouping by functionality 4. Matplotlib Used for  generation of simple and powerful visual...

Training/Dev/Test Set what is training set / dev set / test set In traditional methodology/ when we have small size data we can take 60-20-20 ratio to get training set-validation set/dev set -test set. Now, when we have big data it is fine that the dev set or test set to be less than 10 or 20 percent of your data. Or even 98-1-1 ratio is also fine. One rule of thumb is : Test set and Dev set should come from same distribution. Bias and Variance Bias means the high error rate in training. I may be due to underfitting. For this we can change neural network architecture like network size and number of iterations. Varaince means error rate in Dev set . This may be due to Over fitting of the data . This can be avoided by increasing number of data and regularization. Bias - Variance trade off means balancing both without the increase in other. Regularization is used to reduce the variance . It may hurt bias and bias may increase a little but not much if we have  a bigger network...

Week One - Video One - Why ML STrategy Why we should learn care about ML Strategy Here when we try to improve the performance of the system we should consider about a lot of things . They are: -Amount of data - Amount of diverse data - Train algorithm longer with gradient descent -use another optimization algorithm like Adam -  use bigger network or smaller network depending out requirement -  use drop out - add l2 regularization - network architecture parameters like number of hidden units, Activation function etc. Second Video - Orthogonalization Orthogonalization means in a deep learning network we can change/tune so many things for eg. hyper parameters to get a more performance in the network . So most effective people know what to tune in order to achieve a particular effect. For every set of problem there is a separate solution. Don't mix up the problems and solutions. For that, first we should find out where is the problem , whether it is with training ...

import numpy as np #import tensorflow as tf import os from random import shuffle import cv2 import matlab from tqdm import tqdm import dicom as pdicom from glob import glob from mpl_toolkits.mplot3d.art3d import Poly3DCollection import scipy.ndimage from skimage import morphology from skimage import measure from skimage.transform import resize from sklearn.cluster import KMeans from plotly import __version__ from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot from plotly.tools import FigureFactory as FF from plotly.graph_objs import * import matplotlib.pyplot as plt init_notebook_mode( connected = True ) TRAIN_DIR = '/home/naima.v/mc/CancerImages/Calc_Labelled_Train1' TEST_DIR = '/home/naima.v/mc/CancerImages/Calc_Labelled_Test1' IMG_SIZE = 50 LR = 1e-3 MODEL_NAME = 'CANCERDET-{}-{}.model2' .format(LR, '6conv-basic' ) def readDCMImg(path): g = glob(path + '/*.dcm' ) #print ("Total of %d DICOM images.\nFir...