Big Data analytics Assignment

Big Data Analytics Assessment 2
Overview
Write Python code (especially PySpark where possible) to implement the tasks below (Please explain
your code in detail in your report). Where appropriate, the output of code execution should be presented
that relates to your answer to the tasks into your report, as the evidence of working program (e.g.,
screenshots). Output included without supporting explanation or interpretation will not receive credit.
Where the tasks involve big data analytics with machine learning, you may need to formulate technical
solution step by step, e.g., choose appropriate machine learning models / algorithms, justifying
appropriateness of models/algorithms/techniques used, applying them in the context of the given task,
and practising data visualisation techniques where appropriate. Test your solution and conduct
experiments where appropriate. Evaluate the performance of implemented solution and analyse results
where appropriate. Delve into deep technical explanations of the results and suggest possible
improvement, etc. Make conclusions where appropriate.
You need to present your solution to the tasks into a technical report. The report should be submitted
onto Blackboard.
Assignment tasks
Part (I)
You will need to download the data file “Medical_info.csv” from the Blackboard. The dataset is about a
study of factors predicting serum testosterone levels in a screened population in some country. The key
to the variables in the dataset is explained in the table below. Please note there are null or missing
values in some columns of the dataset.
id Identification Number
age Age (years)
BMI Body Mass Index (kg/m2)
PSA Prostate Specific Antigen (ng/mL)
TG Serum Triglyceride (mg/dL)
Cholesterol Serum Cholesterol (mg/dL)
LDLChole Serum Low-Density Lipoprotein (mg/dL)
HDLChole Serum High-Density Lipoprotein (mg/dL)
Glucose Fasting Plasma Glucose (mg/dL)
Testosterone Serum Testosterone (ng/mL)
BP_1 Blood pressure category, coded as 1=Low, 2=High

  1. Load the data file into a Spark DataFrame (1
    st DataFrame). Describe the structure of the DataFrame.
    (3 marks)
  2. Create a new DataFrame (2
    nd DataFrame) by removing all the rows with null/missing values in the
    1
    st DataFrame and calculate the number of rows removed. (3 marks)
  3. Calculate summary statistics of the ‘age’ feature in the 2
    nd DataFrame, including its min value, max
    value, mean value, median value, variance and standard deviation. Generate a histogram for the
    ‘age’ feature and describe the distribution of the feature. (3 marks)
  4. Display the quartile info of the ‘BMI’ feature in the 2
    nd DataFrame. Generate a boxplot for the ‘BMI’
    feature and discuss the distribution of the feature based on the boxplot. (3 marks)
  5. Use Spark DataFrame API (i.e., expression methods) to count the number of rows where ‘age’ is
    greater than 50 and ‘BP_1’ equals 1. (3 marks)
  6. Use the ‘BP_1’ feature in the 2
    nd DataFrame as the target label, to build two classification models
    based on all other columns as predictors. Conduct performance evaluation for the two models and
    make conclusions. (15 marks)
    Part (II)
    You will need to download the data file “Region_info.csv” from the Blackboard. The dataset provides
    information such as population size, average life expectancy, GDP per capita, and so on for some
    regions. The key to the variables in the dataset is explained in the table below.
    Population Identification Number
    fertility Average number of children a woman in a given country gives birth to
    HIV HIV infection rate
    CO2 CO2 emission (tonnes per person)
    BMI_male Body Mass Index (male)
    GDP Gross Domestic Product
    BMI_female Body Mass Index (female)
    life Average life expectancy
    child_mortality Death of children under 5 years of age per 1000 live births
    region Region
  7. Load the data file into a Spark DataFrame (1st DataFrame). Describe the structure of the created
    data frame. (3 marks)
  8. Create a new DataFrame (2
    nd DataFrame) by removing the ‘region’ column. (3 marks)
  9. Use a graph, explore and describe the relationship between ‘fertility’ feature and ‘life’ feature in the
    2
    nd DataFrame. (3 marks)
  10. Use Spark SQL query to display the ‘fertility’ and ‘life’ columns in the 2nd DataFrame where ‘fertility’
    is great than 1.0 and ‘life’ is greater than 70. (3 marks)
  11. Build a linear regression model to predict life expectancy (the ‘life’ column) in the 2nd DataFrame
    using the ‘fertility’ column as the predictor. Conduct performance evaluation for the model and make
    conclusions. (9 marks)
  12. Build a Lasso regression model to predict life expectancy (the ‘life column) in the 2nd Data Frame
    using all other columns as the predictor. Conduct performance evaluation for the model and make
    conclusions. (9 marks)