Dora the Techplorer

Embed JSX code with if condition in React app

2020-08-01T20:13:00.001+08:00

renderPrice(pt) {

if (pt.amountpaid!="") {

let ctrl = (

<div>

<label>Price</label><br/>

</div>

</>

);

return ctrl;

} //end if

} //end renderPrice

render() {

    const priceElement = this.renderPrice(purchasetransaction);

    return (
                <div style={{marginTop: '5em',marginLeft:'2em'}}>
                   
                    {priceElement}

                    <p></p>
                   </div>
        )

    } //end render

Reset root password on MySQL on Windows 10 laptop

2020-07-28T15:23:00.002+08:00

Stop SQL Server

Create a text file "mysql-init.txt" in C:\temp directory and paste in this content

ALTER USER 'root'@'localhost' IDENTIFIED BY 'MyNewPass';

Open a command prompt window and run this command

mysqld --defaults-file="C:\\ProgramData\\MySQL\\MySQL Server 8.0\\my.ini" --init-file="C:\\temp\\mysql-init.txt"

Create a SQL table with autoincrement id and timestamps

2020-07-28T15:05:00.000+08:00

CREATE TABLE `tutorials` (

`id` int(11) NOT NULL AUTO_INCREMENT,

`title` varchar(255) DEFAULT NULL,

`description` varchar(255) DEFAULT NULL,

`published` tinyint(1) DEFAULT '1',

`createdAt` timestamp NULL DEFAULT CURRENT_TIMESTAMP,

`updatedAt` timestamp NULL DEFAULT NULL ON UPDATE CURRENT_TIMESTAMP,

PRIMARY KEY (`id`)

)

AWS Policy editor

2019-05-26T22:23:00.001+08:00

https://awspolicygen.s3.amazonaws.com/policygen.html

Download files from Databricks Community Edition

2019-05-02T10:33:00.000+08:00

Syntax is:

https://community.cloud.databricks.com/files/myStuffs/yourdata.csv/part-00000?o=<yournumber>

Run pyspark on Juypter notebook

2019-04-17T10:44:00.001+08:00

$which python
$whereis python

# install EPEL repository first
$ sudo yum install epel-release
# install python-pip
$ sudo yum -y install python-pip

sudo pip install --upgrade setuptools

wget https://repo.anaconda.com/archive/Anaconda2-5.0.1-Linux-x86_64.sh

sudo sh Anaconda2-5.0.1-Linux-x86_64.sh

1) Install PySpark
pip install pyspark

2) Install Java

3) Install Jupyter notebook
pip install jupyter

4) Install find
pip install findspark

%env SPARK_HOME=c:\spark

# To find out where the pyspark
import findspark
findspark.init()

# Creating Spark Context
from pyspark import SparkContext
sc = SparkContext("local", "first app")

# Calculating words count
text_file = sc.textFile("OneSentence.txt")
counts = text_file.flatMap(lambda line: line.split(" ")) \
.map(lambda word: (word, 1)) \
.reduceByKey(lambda a, b: a + b)

# Printing each word with its respective count
output = counts.collect()
for (word, count) in output:
print("{}: {}".format(word, count))

# Stopping Spark Context
sc.stop()

Install Python 3.5 on Centos 6

sudo yum install centos-release-scl
sudo yum info rh-python35
sudo yum install rh-python35
sudo scl enable rh-python35 bash

https://www.2daygeek.com/3-methods-to-install-latest-python3-package-on-centos-6-system/

Useful icons

2019-04-16T09:46:00.002+08:00

https://pngtree.com/free-icon/node-other-database-cluster_741016
https://www.clipartmax.com/

Set up Cloudera CDH 5.13 todo

2019-04-08T16:26:00.002+08:00

sudo /etc/resolv.conf

Edit Virtual Box forwarding, add port 22

Troubleshooting Hive

2019-04-08T13:31:00.003+08:00

https://www.cloudera.com/documentation/enterprise/5-14-x/topics/cdh_ig_hiveserver2_start_stop.html

Check if Hive metastore is running
sudo service hive-metastore status
sudo service hive-metastore stop

sudo service hive-metastore start
sudo service hive-metastore restart

sudo service hive-server2 stop

Check if Hive Server2 is running
$ sudo service hive-server2 status

Start HiveServer2:
$ sudo service hive-server2 start

To stop HiveServer2:
$ sudo service hive-server2 stop

Check Hive warehouse
hadoop fs -ls /user/hive/warehouse

Connect using Beeline
beeline -u jdbc:hive2://
!connect -u jdbc:hive2://

Login Hortonworks Sandbox with WinSCP

2019-04-03T15:39:00.000+08:00

ssh root@sandbox-hdp.hortonworks.com -p 2222

First-time login, use hadoop as password
You will be asked to change password upon first login

C:\Users\<username>\.ssh

kill yum process

2019-04-03T15:11:00.002+08:00

yum list
ps -ef | grep 13023
kill -9 13023

Troubleshoot DNS on CDH

2019-04-02T11:39:00.001+08:00

sudo nano /etc/resolv.conf

nameserver 8.8.8.8

# /etc/init.d/network stop
# /etc/init.d/network start

sudo systemctl restart network

https://www.cyberciti.biz/faq/linux-restart-network-interface/

Download zip file using Python

2019-03-04T14:44:00.002+08:00

https://www.zerotosingularity.com/blog/downloading-datasets-introducting-pdl/

import requests
import zipfile
import os

# download file
resp = requests.get(f"{url}{file}", allow_redirects=True, stream=True)

filename = f"{download_dir}{file}"

zfile = open(filename, 'wb')
zfile.write(resp.content)
zfile.close()

zipf = zipfile.ZipFile(filename, 'r')
zipf.extractall(download_dir)
zipf.close()

os.remove(filename)

YEAR=2015
for MONTH=`seq -w 1 12`; do
PARAMS="UserTableName...
RESPONSE=$(curl -X POST --data "$PARAMS"
http://www.transtats.bts.gov/DownLoad_Table.asp?Table_ID=236&Has_Group=3&Is_
Zipped=0)
echo "Received $RESPONSE"
ZIPFILE=$(echo $RESPONSE | tr '\"' '\n' | grep zip)
echo $ZIPFILE
curl -o $YEAR$MONTH.zip $ZIPFILE
done

Is this useful?
https://stackoverflow.com/questions/51665087/python-download-zip-file-from-restapi

Get a book from Gutenberg

2019-03-01T15:44:00.001+08:00

wget http://www.gutenberg.org/cache/epub/11/pg11.txt

utorial: Using pandas with Large Data Sets

2019-01-30T14:57:00.002+08:00

https://www.dataquest.io/blog/pandas-big-data/

Adjust width of pandas dataframe

2019-01-30T14:56:00.000+08:00

https://stackoverflow.com/questions/52369572/python-how-to-get-data-types-for-all-columns-in-csv-file

pd.set_option('max_info_columns', 200)

Create a Python 3.6 environment on top of Python 3.7

2019-01-29T20:57:00.003+08:00

cd d:\
python -m venv D:\mypython36_venv
rem cd d:\mypython36_venv
D:\mypython36_venv\Scripts\activate
D:\mypython36_venv\Scripts\deactivate

This one is for Python 2.7 on Raspberry Pi
python -m virtualenv python_venv
cd python_venv
source mypython/bin/activate

Find out where is mysql data directory

2019-01-24T14:52:00.003+08:00

'SHOW VARIABLES WHERE Variable_Name LIKE "%dir"

'SHOW VARIABLES WHERE Variable_Name = "datadir"'

datasets

2019-01-23T17:04:00.002+08:00

https://www.chicago.gov/city/en/depts/dhr/dataset/current_employeenamessalariesandpositiontitles.html

Sourced from: https://janakiev.com/blog/pandas-groupby/

check if ambari is ready

2019-01-16T22:31:00.001+08:00

https://ambari.apache.org/1.2.0/installing-hadoop-using-ambari/content/ambari-chap2-3.html\

3. Start the Ambari Server
To start the Ambari Server:

ambari-server start
To check the Ambari Server processes:

ps -ef | grep Ambari
To stop the Ambari Server:

ambari-server stop

https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Shell

Install mysql on hortonworks sandbox

2019-01-01T22:24:00.002+08:00

https://community.hortonworks.com/questions/78566/where-is-mysql-in-hdp-25.html

https://community.hortonworks.com/questions/203206/mysql-default-password-first-time-sandbox-login.html

Transfer files from Windows laptop to the Hortonworks VM

2019-01-01T20:32:00.002+08:00

On Windows laptop, open a new terminal window, navigate to the folder where you have stored your files.
Execute the following commands to transfer the files from your laptop to your Hortonworks VM

scp -P 2222 * filetotransfer.txt root@sandbox-hdp.hortonworks.com:/tmp/data

Hortonworks Settings to change

2019-01-01T14:15:00.001+08:00

Minimum replicated blocks %

Change to zero

This will prevent HDFS from going into safe mode, which will in turn cause YARN not to run properly

YARN service starting problem

2019-01-01T14:08:00.002+08:00

Check if HDFS is in safe mode

hdfs dfsadmin -safemode get

Force HDFS to leave safe mode

hdfs dfsadmin -safemode leave

HDFS safemode command

hdfs dfsadmin -safemode [enter | leave | get]

Install GUI on Debian stretch

2018-12-22T22:55:00.002+08:00

apt-get install task-gnome-desktop