Author: Tom
R – Četnost podruhé
Mějme následující data:
|
1 2 3 4 5 6 7 8 |
> head(dotaznik.csv[,c(2,3)]) type gender 1 H F 2 H M 3 A F 4 A M 5 A F 6 H M |
A potřebujeme zjistit absolutní a relativní četnost výskytu žen a mužů, rozdělenou ještě podle hodnoty v type .
|
1 2 3 4 5 6 |
> tbl <- table(dotaznik.csv$gender, dotaznik.csv$type) > tbl A H M 18 16 Ž 17 19 |
Opět převedeme na data frame
|
1 2 3 4 5 6 7 |
> tbl.frame <- as.data.frame(tbl) > tbl.frame Var1 Var2 Freq 1 M A 18 2 Ž A 17 3 M H 16 4 Ž H 19 |
A máme absolutní četnost. Teď vypočítáme relativní četnost.
|
1 2 3 4 5 6 7 |
> library(plyr) > ddply(tbl.frame, .(Var2), transform, prop=Freq/sum(Freq)) Var1 Var2 Freq prop 1 M A 18 0.5142857 2 Ž A 17 0.4857143 3 M H 16 0.4571429 4 Ž H 19 0.5428571 |
Což znamená: pro každý subset z framu tbl.frame , rozdělený podle proměnné Var2 , proveď transform , a vypočti relativní četnost podle frekvence (abs. četnosti) a součtu hodnot frekvence.
R – četnost a graf
Dostal jsem pověření z nejvyšších míst vyrobit statistické zhodnocení dotazníků. A protože bytostně nemám rád Excel, našel jsem R a zkouším.
Udělal jsem data do CSV souboru a nahrál do R (RStudio)
|
1 2 3 4 5 6 7 8 |
> head(dotaznik.csv) num type gender age height weight edu 1 1 H M 60 182 100 Vysokoškolské 2 2 H M 49 188 102 Vyučen/a 3 3 H M 61 176 75 Vyučen/a 4 4 H M 56 180 110 Střední 5 5 H M 47 180 95 Vysokoškolské 6 6 H M 48 178 95 Střední |
Zkusíme vyrobit četnost vzdělání (edu) a potom ji ještě rozdělit podle pohlaví (gender).
|
1 2 3 4 5 6 7 |
> edu <- table(dotaznik.csv$edu) > edu Střední Vysokoškolské Vyšší odborné Vyučen/a 30 7 2 25 Základní 6 |
Máme tabulku, ale potřebujeme z ní frame
|
1 2 3 4 5 6 7 8 |
> fedu <- as.data.frame(edu) > fedu Var1 Freq 1 Střední 30 2 Vysokoškolské 7 3 Vyšší odborné 2 4 Vyučen/a 25 5 Základní 6 |
Přidáme relativní četnost (mean)
|
1 2 3 4 5 6 7 8 |
> fedu$mean <- fedu$Freq / sum(fedu$Freq) > fedu Var1 Freq mean 1 Střední 30 0.42857143 2 Vysokoškolské 7 0.10000000 3 Vyšší odborné 2 0.02857143 4 Vyučen/a 25 0.35714286 5 Základní 6 0.08571429 |
A teď na graf. Používáme ggplot2.
|
1 |
ggplot(fedu, aes(x=Var1, y=Freq)) + geom_bar(stat='identity') |
Fajn. Ale chceme graf otočit
|
1 |
ggplot(fedu, aes(x=Var1, y=Freq)) + geom_bar(stat='identity') + coord_flip() |
Teď ještě smazat popisek osy x a přepsat osu y.
|
1 2 3 4 5 |
ggplot(fedu, aes(x=Var1, y=Freq)) + geom_bar(stat='identity') + coord_flip() + xlab('Vzdělání') + theme(axis.title.x = element_blank()) + ggtitle('Dosažené vzdělání') |
A finálně přidáme hodnoty frekvence do jednotlivých sloupců.
|
1 2 3 4 5 6 7 |
ggplot(fedu, aes(x=Var1, y=Freq)) + geom_bar(stat='identity', position=position_dodge()) + geom_text(aes(label=Freq), hjust=1.3, color="white",position = position_dodge(0.9), size=3.5) + coord_flip() + xlab('Vzdělání') + theme(axis.title.x = element_blank()) + ggtitle('Dosažené vzdělání') |
A teď trochu komplexněji. Ještě to rozdělíme na muže a ženy..
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
> edu <- table(dotaznik.csv$gender, dotaznik.csv$edu) > edu Střední Vysokoškolské Vyšší odborné Vyučen/a Základní M 10 4 2 17 1 Ž 20 3 0 8 5 > fedu <- as.data.frame(edu) > fedu Var1 Var2 Freq 1 M Střední 10 2 Ž Střední 20 3 M Vysokoškolské 4 4 Ž Vysokoškolské 3 5 M Vyšší odborné 2 6 Ž Vyšší odborné 0 7 M Vyučen/a 17 8 Ž Vyučen/a 8 9 M Základní 1 10 Ž Základní 5 |
a přidáme graf
|
1 2 3 4 5 6 7 8 |
> ggplot(fedu, aes(x=Var2, y=Freq, fill=Var1)) + geom_bar(stat='identity', position=position_dodge()) + geom_text(aes(label=Freq), hjust=1.6, color="white",position = position_dodge(0.9), size=3.5) + coord_flip() + xlab('Vzdělání') + theme(axis.title.x = element_blank()) + ggtitle('Dosažené vzdělání') + labs(fill = 'Pohlaví') |
A co když budeme chtít jeden bar, ale rozdělený podle hodnot?
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
# seridime podle Var2 a Freq > library(plyr) > fedu_s <- arrange(fedu, Var2, Freq) > fedu_s Var1 Var2 Freq 1 M Střední 10 2 Ž Střední 20 3 Ž Vysokoškolské 3 4 M Vysokoškolské 4 5 Ž Vyšší odborné 0 6 M Vyšší odborné 2 7 Ž Vyučen/a 8 8 M Vyučen/a 17 9 M Základní 1 10 Ž Základní 5 # a pridame souhrnny soucet, rozdeleny s kazdym jinym Var2 > fedu_s_sum <- ddply(fedu_s, 'Var2', transform, label_ypos=cumsum(Freq)) > fedu_s_sum Var1 Var2 Freq label_ypos 1 M Střední 10 10 2 Ž Střední 20 30 3 Ž Vysokoškolské 3 3 4 M Vysokoškolské 4 7 5 Ž Vyšší odborné 0 0 6 M Vyšší odborné 2 2 7 Ž Vyučen/a 8 8 8 M Vyučen/a 17 25 9 M Základní 1 1 10 Ž Základní 5 6 |
A graf
|
1 2 3 4 5 6 7 8 |
ggplot(fedu_s_sum, aes(x=Var2, y=Freq, fill=Var1)) + geom_bar(stat='identity') + geom_text(aes(label=Freq, y=label_ypos), hjust=1.6, color="white", size=3.5) + coord_flip() + xlab('Vzdělání') + theme(axis.title.x = element_blank()) + ggtitle('Dosažené vzdělání') + labs(fill = 'Pohlaví') |
STM32F4Discovery + eLua + OSX
I fixed few bugs when compiling elua from git on osx – clone my branch clone official repo here.
|
1 |
$ ./build_elua.lua board=stm32f4discovery |
create .bin file
|
1 |
$ arm-none-eabi-objcopy -O binary elua_lua_stm32f4discovery.elf elua_lua_stm32f4discovery.bin |
create openocd config file for stm32f4 board:
|
1 2 3 4 5 6 |
$ cat ~/stm32f4discovery.cfg # stm32f4discover board source [find interface/stlink-v2-1.cfg] transport select hla_swd source [find target/stm32f4x.cfg] reset_config srst_only |
and upload to the board:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
$ openocd -f ~/stm32f4discovery.cfg \ -c "init" \ -c "reset halt" \ -c "sleep 100" \ -c "wait_halt 2" -c "echo \"--- Writing elua_lua_stm32f4discovery.bin\"" \ -c "flash write_image erase elua_lua_stm32f4discovery.bin 0x08000000" \ -c "sleep 100" \ -c "echo \"--- Verifying\"" \ -c "verify_image elua_lua_stm32f4discovery.bin 0x08000000" \ -c "sleep 100" \ -c "echo \"--- Done\"" \ -c "resume" \ -c "shutdown" Open On-Chip Debugger 0.9.0 (2015-11-16-01:48) Licensed under GNU GPL v2 For bug reports, read http://openocd.org/doc/doxygen/bugs.html Info : The selected transport took over low-level target control. The results might differ compared to plain JTAG/SWD adapter speed: 2000 kHz adapter_nsrst_delay: 100 none separate srst_only separate srst_nogate srst_open_drain connect_deassert_srst Info : Unable to match requested speed 2000 kHz, using 1800 kHz Info : Unable to match requested speed 2000 kHz, using 1800 kHz Info : clock speed 1800 kHz Info : STLINK v2 JTAG v25 API v2 SWIM v14 VID 0x0483 PID 0x374B Info : using stlink api v2 Info : Target voltage: 2.884520 Info : stm32f4x.cpu: hardware has 6 breakpoints, 4 watchpoints target state: halted target halted due to debug-request, current mode: Thread xPSR: 0x01000000 pc: 0x080063f0 msp: 0x20000c78 --- Writing elua_lua_stm32f4discovery.bin auto erase enabled Info : device id = 0x10076413 Info : flash size = 1024kbytes wrote 262144 bytes from file elua_lua_stm32f4discovery.bin in 7.783807s (32.889 KiB/s) --- Verifying target state: halted target halted due to breakpoint, current mode: Thread xPSR: 0x61000000 pc: 0x2000002e msp: 0x20000c78 verified 220776 bytes in 2.229936s (96.685 KiB/s) --- Done shutdown command invoked |
Now you can connect to the board using /dev/tty.usbmodem621, 115200/8/n/1. You should get this prompt:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
eLua# eLua# help Shell commands: help - shell help lua - start a Lua session ls - lists files and directories dir - lists files and directories cat - list the contents of a file type - list the contents of a file recv - receive files via XMODEM cp - copy files mv - move/rename files rm - remove files ver - show version information mkdir - create directories exit - exit the shell For more information use 'help <command>'. eLua# lua Press CTRL+Z to exit Lua Lua 5.1.4 Copyright (C) 1994-2011 Lua.org, PUC-Rio > print(pd.board() .. "/" .. pd.platform() .. "/" .. pd.cpu()) STM32F4DISCOVERY/STM32F4/STM32F407VG |
enjoy :)
nodemcu – simple webserver
I’ve modified and updated this source:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 |
local SSID = "my_wifi_ssid" local SSID_PASSWORD = "my_wifi_password" local function http_header(conn) conn:send('HTTP/1.1 200 OK\n\n') conn:send('<!DOCTYPE HTML>\n') conn:send('<html>\n') conn:send('<head><meta content="text/html; charset=utf-8">\n') conn:send('<title>ESP8266 znouza test</title></head>\n') end local function connect (conn, data) local query_data conn:on ("receive", function (cn, req_data) query_data = get_http_req (req_data) print (query_data["METHOD"] .. " " .. " " .. query_data["User-Agent"]) http_header(cn) cn:send ("<body>") cn:send ("<h1>Hello World from ESP8266 and NodeMCU!!</h1>") cn:send ("</body></html>") -- Close the connection for the request cn:close ( ) end) end function wait_for_wifi_conn ( ) tmr.alarm (1, 1000, 1, function ( ) if wifi.sta.getip ( ) == nil then print ("Waiting for Wifi connection") else tmr.stop (1) print ("ESP8266 mode is: " .. wifi.getmode ( )) print ("The module MAC address is: " .. wifi.sta.getmac ( )) print ("Config done, IP is " .. wifi.sta.getip ( )) end end) end -- Build and return a table of the http request data function get_http_req (instr) local t = {} local first = nil local key, v, strt_ndx, end_ndx for str in string.gmatch (instr, "([^\n]+)") do -- First line in the method and path if (first == nil) then first = 1 strt_ndx, end_ndx = string.find (str, "([^ ]+)") v = trim (string.sub (str, end_ndx + 2)) key = trim (string.sub (str, strt_ndx, end_ndx)) t["METHOD"] = key t["REQUEST"] = v else -- Process and remaining ":" fields strt_ndx, end_ndx = string.find (str, "([^:]+)") if (end_ndx ~= nil) then v = trim (string.sub (str, end_ndx + 2)) key = trim (string.sub (str, strt_ndx, end_ndx)) t[key] = v end end end return t end -- String trim left and right function trim (s) return (s:gsub ("^%s*(.-)%s*$", "%1")) end -- Configure the ESP as a station (client) wifi.setmode (wifi.STATION) wifi.sta.config (SSID, SSID_PASSWORD,1) -- Hang out until we get a wifi connection before the httpd server is started. wait_for_wifi_conn ( ) -- Create the httpd server svr = net.createServer (net.TCP, 30) -- Server listening on port 80, call connect function if a request is received svr:listen (80, connect) |
nodemcu + el capitan
- download and install CH340g driver (check your FTDI chip)
- install CoolTerm (http://freeware.the-meiers.org/)
- install ESPlorer (http://esp8266.ru/esplorer/)
- open CoolTerm, options, choose appropriate /dev/cu.* port and connect
To flash
- download firmware
- Install esptool
|
1 2 3 |
git clone https://github.com/themadinventor/esptool.git cd esptool sudo python ./setup.py install |
- upgrade firmware
|
1 2 3 4 5 6 7 8 |
$ python ./esptool.py --port=/dev/cu.wchusbserial620 write_flash -fm=dio -fs=32m 0x00000 /tmp/nodemcu_integer_0.9.6-dev_20150704.bin Connecting... Erasing flash... Took 3.92s to erase flash block Wrote 450560 bytes at 0x00000000 in 54.1 seconds (66.6 kbit/s)... Leaving... |
iPhone5 power button repaired
Common issue. Power button got broken after a while, I switched to iPhone6, so this ‘old’ phone stayed forgotten. Then my son wanted to use it, so I had to repair it. Looked around, for the first time it as eligible to Apple repair program, but after a while my SN was removed, not sure why. Anyway, the off-warranty repair costs more than 2500CZK (w/o VAT), quite a lot. So I looked around and found few links. Ordered spare button/ribbon cable, few tools and .. got it repaired (but I have to admit, it was a tough fight) :)
Price to repair was (excluding tools) around $4.
Things ordered from Aliexpress:
- screwdriver (philips etc) with small bits
- spudger tool
- iSesamo opening tool
- switch flex table [spare part only needed]
Cool links to follow when repairing
PS: a small advice at the end. Sort your screws to small groups, on some sticker or a stick tape, put them in that order as removed – as there’s a bunch of really small screws and almost every one differs from others
Side by side images
Quick script. As I’ve got FLIR images of my house, I wanted to have them side by side – original image + FLIR. So I wrote some small (fast) script in bash to do that.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 |
#!/bin/bash ir=IR_ img=DC_ im_start=$1 ir_start=$((im_start-1)) echo "Starting from $start - $(printf "%s%04d.jpg" $img $im_start)" echo "IR will be from $ir_start - $(printf "%s%04d.jpg" $ir $ir_start)" if [[ ! -f "$(printf "%s%04d.jpg" $img $im_start)" ]]; then echo "Image doesn't exist!" exit fi if [[ ! -f "$(printf "%s%04d.jpg" $ir $ir_start)" ]]; then echo "IR doesn't exist!" exit fi loop=1 while [ $loop -ne 0 ];do image=$(printf "%s%04d.jpg" $img $im_start) ir_image=$(printf "%s%04d.jpg" $ir $ir_start) combined_image=$(printf "flir-%04d-%04d.jpg" $im_start $ir_start) echo -n "$image + $ir_image --> $combined" montage -mode concatenate -tile x1 $image $ir_image $combined_image echo "..Done!" # increase im_start=$((im_start+2)) ir_start=$((ir_start+2)) #echo "Next file $(printf "%s%04d.jpg" $img $im_start)" # test if file? if [[ ! -f "$(printf "%s%04d.jpg" $img $im_start)" ]]; then loop=0 fi done |
And the resulting image:
sidekiq getting stuck?
We had quite strange scenario. Two hosts with sidekiq, one working well, second one gets stuck after few seconds or minutes. Happened in workers with opening new TCP/UDP connections. So, I started to review all the files, where ‘max files’ is set.
|
1 2 3 4 5 6 7 8 9 10 11 |
$ cat /etc/security/limits.conf * soft nofile 30000 * hard nofile 60000 tom@web:~$ ulimit -Hn 60000 tom@web:~$ ulimit -Sn 30000 tom@web:~$ /sbin/sysctl -a | grep "file-max" fs.file-max = 60000 |
So, all this looks OK so far. After numerous checks I’ve found, that our eye process is running for a while, maybe since the beginning and probably it took old file-max settings. How to check? Simply run eye info or get PID of the sidekiq process and run following command (assume 22613 is the PID):
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
tom@web:~$ cat /proc/22613/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 516140 516140 processes Max open files 1024 4096 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 516140 516140 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us |
Gotcha! 1024 as soft-limit, 4096 hard limit. The fix is pretty simple – just quit eye , load it’s config and restart sidekiq.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
tom@web:~$ eye quit quit... tom@web:~$ eye load config.eye eye started! config loaded! tom@web~$ eye restart sidekiq tom@web:~$ cat /proc/4624/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 516140 516140 processes Max open files 30000 60000 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 516140 516140 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us |
voila! We’re back on track :)
sendmail + osx 10.10.4/ios 8.4/ios 9
After upgrade I was unable se to send mails through my sendmail, from both OSX 10.10.4 and upgraded iOS9 …
this was in my sendmail log:
|
1 2 3 |
Jul 12 19:58:03 orin sendmail[12164]: STARTTLS=server, error: accept failed=0, SSL_error=1, errno=0, retry=-1 Jul 12 19:58:03 orin sendmail[12164]: STARTTLS=server: 12164:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handsh ake failure:s3_pkt.c:1053:SSL alert number 40 |
fix is amazingly easy and effective:
generate dh2048 certificate:
|
1 |
openssl gendh -out dh_2048.pem -2 2048 |
and add to sendmail.mc:
|
1 |
define(`confDH_PARAMETERS',`/etc/mail/certs/dh_2048.pem') |
then recompile your .mc and restart sendmail.