SDS-2.2, Scalable Data Science

Archived YouTube video of this live unedited lab-lecture:

Let us visit an interactive visual cognitive tool for the basics ideas in linear regression:

http://setosa.io/ev/ordinary-least-squares-regression/

The following video is a very concise and thorough treatment of linear regression for those who have taken the 200-level linear algebra. Others can fully understand it with some effort and revisiting.

Linear Regression by Ameet Talwalkar in BerkeleyX: CS190.1x Scalable Machine Learning

(watch now 11:13):

Ridge regression has a Bayesian interpretation where the weights have a zero-mean Gaussian prior. See 7.5 in Murphy's Machine Learning: A Probabilistic Perspective for details.

Please take notes in mark-down if you want.

For latex math within markdown you can do the following for in-line maths: $\mathbf{A}_{i,j} \in \mathbb{R}^1$ . And to write maths in display mode do the following:

$\mathbf{A} \in \mathbb{R}^{m \times d}$

You will need to write such notes for your final project presentation!

MillonSongs Ridge Regression by Ameet Talwalkar in BerkeleyX: CS190.1x Scalable Machine Learning

(watch later 7:47):

Covers the training, test and validation and grid search... ridger regression...

Take your own notes if you like.

Gradient Descent by Ameet Talwalkar in BerkeleyX: CS190.1x Scalable Machine Learning

(watch later 11:19):

Please take notes if you want to.

Now stop the stream and restart it to circumvent BerkeleyX CC claims

Getting our hands dirty with Darren Wilkinson's blog on regression

Let's follow the exposition in:

You need to scroll down the fitst link embedded in iframe below to get to the section on Analysis of quantitative data with Descriptive statistics and Linear regression sub-sections (you can skip the earlier part that shows you how to run everything locally in spark-shell - try this on your own later).

<p> <a href="http://spark.apache.org/docs/latest/index.html" target="_blank"> Fallback link for browsers that, unlikely, don't support frames </a> </p>

Let's do this live now...

import breeze.stats.distributions._
def x = Gaussian(1.0,2.0).sample(10000)
val xRdd = sc.parallelize(x)

import breeze.stats.distributions._
x: IndexedSeq[Double]
xRdd: org.apache.spark.rdd.RDD[Double] = ParallelCollectionRDD[3402] at parallelize at <console>:44

println(xRdd.mean)
println(xRdd.sampleVariance)

0.9791454017224025
3.9376502663091917

val xStats = xRdd.stats
xStats.mean
xStats.sampleVariance
xStats.sum

xStats: org.apache.spark.util.StatCounter = (count: 10000, mean: 0.979145, stdev: 1.984252, max: 8.491747, min: -6.160311)
res7: Double = 9791.454017224021

val x2 = Gaussian(0.0,1.0).sample(10000) // 10,000 Gaussian samples with mean 0.0 and std dev 1.0
val xx = x zip x2 // creating tuples { (x,x2)_1, ... , (x,x2)_10000}
val lp = xx map {p => 2.0*p._1 + 1.0*p._2 + 1.5} // { lp_i := (2.0*x + 1.0 * x2 + 1.5)_i, i=1,...,10000 }
val eps = Gaussian(0.0,1.0).sample(10000) // standrad normal errors
val y = (lp zip eps) map (p => p._1 + p._2)
val yx = (y zip xx) map (p => (p._1, p._2._1, p._2._2))

x2: IndexedSeq[Double] = Vector(1.7180200627538347, 0.7612579495121726, 1.1044351189222115, -1.0458887821837508, 0.49496877204701656, -0.7877659459352377, 2.2839982579960774, 2.0904918533649792, -0.3154431840702555, 1.08225194191406, -0.08599324189813665, 2.038160537801021, 1.129108822212312, 0.0332092866052016, -0.8484628252915802, 1.7489986908075437, 0.5905753508188052, 0.4532853262146371, 2.1071390618861483, -2.4333190509254523, 1.8666210820194409, -0.4716364393080115, 0.43215718847354717, -0.0508145229262295, 0.2318912449411393, 0.6642118433489868, -0.34188724147579436, -0.955949132880485, 1.0374062592965703, 0.30231115166753975, -0.7053596883786023, 0.07772257556676218, 0.2734856951677595, -1.1171292063497231, -2.25342400569796, 0.8662107078313865, -1.2219562085123437, -0.049916676934831865, -0.17692737947420795, 0.023142072745985356, 0.23206737522014134, -0.24576353385861008, -0.2650095664570966, 1.0431508512261574, 1.3426415201672703, -0.3411574543509265, -0.05605340120237845, 0.8999076240632354, 2.0209749614041304, -0.6983443417434481, -0.5973195209029522, 0.856908070217254, -1.1828737859192346, 1.510478238660708, -2.2247519539098066, -1.5582965573761123, 0.06906490685773707, -2.042771261983173, -1.2655530046980348, 0.7776369909128293, 0.3805465428130765, 0.0037395890916070047, -0.44857098618012636, -0.3839517041971513, 0.8637075914602533, 1.9008996217205008, -0.9732951219651634, 0.47808650335049846, 0.7810581695915911, 1.9073405712038916, 0.6155484840550374, 1.285720743143111, -0.15563461998811107, 1.1963614398365867, -0.3107393094384793, 0.1442507745437628, -1.0062706124372303, -3.2144116143213446, -0.3791151897202505, -1.1018272395002096, 0.014112001386570377, 1.417768685115442, 1.849946357990971, 2.297256967452079, 2.863967849863008, 1.1727349692923696, 0.2638734749999953, -0.6386146927254167, -0.7621438861307569, -0.1761694299607887, -1.0034798540546024, -0.5624664952260037, 0.2742520833159514, -1.4493854639080737, -1.5212897485964094, 1.4720420607751994, -0.2291161453735276, -1.7256651550846345, -0.26044179729408784, -0.2508065280088541, 0.8772202042323987, 1.2718923272585114, 1.2727351580988864, 0.07159473932867604, -0.05641516601493637, -1.3063232275399193, -1.0188776063643208, 0.20177767101494343, -0.19326082514830212, 0.46880870506107236, 0.02644986556162372, -1.66251624836091, -0.6563531547897851, 0.18965199488691234, 0.8733631401865725, -2.624790720525607, 0.8857326290925008, 1.4816882795472734, -0.9975488820685854, 0.6481986514102127, -0.8749709644202547, 0.7579070844724433, 0.2622465401074379, 0.6746683513550983, -1.263448996919519, -0.7381981835849909, 0.14994282623171254, 0.15669755947388275, 1.520118672383452, 0.5801060842383697, 1.0661532398756548, 1.9130703293252176, 1.1358559752636856, -0.08681707315553623, -0.0018107786449524216, 0.4046562361147712, 3.3060952703417024, 1.592776279249439, 0.7656314587366151, -0.9269798956639591, 0.39262906647553725, -0.1179070920628396, -0.2398427544759008, -0.8805376802960007, 1.185098287008337, 2.127758749533745, -1.6193304015036736, 0.389753475903186, -0.6320665313267828, -0.6071489604350191, -0.17176036495514585, 2.3082425701198837, -1.0924630230651688, -0.13377939063602365, 0.19472876609435377, -0.6985150412597225, -0.30183923871602275, 0.29017288546487807, -0.08506161946062703, 0.7880599126205807, -0.4152095745664085, -0.5708545974042495, 0.3683149142802798, 0.007561875612265056, -0.7047584310186249, 0.8205055007859963, -0.45649764363696793, 0.14093204594566655, -0.6566752621490228, 0.6637207009154089, 2.145603957540527, -0.027836928631156135, 0.7423557406071833, 0.2149356026948634, 1.2789556634519788, -0.421745799494621, -0.624867593419885, 1.3617530989714166, 0.3786873316449697, 0.294583494801177, -0.4244208241183847, 0.37273978804861935, -2.7888200870508477, 0.13090423295330372, 1.4662350241059718, 0.6482747044416928, 0.4099329018255514, -0.9052966951952968, -0.7940755000771325, -0.24743077206220485, 0.1052809704647746, 0.7135031861304673, 0.3890552443985578, -1.0205843406492123, 0.9951670813851188, 0.05125608850433839, -0.764826774556313, 0.6698230705476478, -0.6000009790234297, -0.44656809427334865, 0.708775810179257, -0.20972725637246492, -2.0064825743727734, -0.6445426124434558, -1.1342463969194072, -0.754557781468656, 1.486456873017418, 0.07209103107719902, -1.6002365728809387, -1.046111170467302, 1.5026911880047211, -0.1457843644663685, -0.13947748839418284, 0.42851687041940495, -0.3451690864154465, -0.982541496031045, 1.0986738175653772, 0.6994670164774653, 1.4268673094155087, 1.1377152102285677, 1.681158675564279, -1.0243192501461797, 0.7157476281459816, 0.1987225329603131, -0.8995597231470148, -0.6454347709276017, -1.210748247776176, -1.2029648168386262, -0.5850933305769891, 0.20655562292795113, 0.19367807044160337, -1.9225118432641704, -0.3891688523373755, 0.750778001298227, -1.4563569125990905, 0.32178951363534475, -0.42696414784080483, -0.5745967327557552, -0.33987806460763226, -1.2201444717297956, -0.4804118001724722, 0.08573017271990857, 1.2739693849684661, -0.6860805257029206, -0.12149128270027325, -0.13213792842778785, 2.1931880748351817, -1.4266518171274647, -1.2862151522144258, -1.3894824574835245, 0.42483399623760804, 1.8191505956546687, 1.3970685728640833, 0.9731809955429377, 0.5893405158842995, -0.3828769935559573, 1.0564581904685673, 1.2069544831918129, -0.35225447445437746, 0.980075180240185, -0.2214099969048667, -0.7765925304274375, -0.40713952940074355, -0.3220702027858841, -0.7156418098061273, -0.8797952571244195, 0.711322217928678, -1.4874834705020474, 0.26569947209899103, -0.6733470729032577, 0.3447471414040174, 1.2318617256455997, 1.44689615341149, 0.8105317859479295, 1.558192810324353, 1.296473628465781, -0.3827008454322967, -0.8097542023433437, 1.5251085260269759, 0.6212940140441159, 0.5407362869930761, 0.41821399753599575, -0.8079238389605706, -0.9085463275200037, -0.6225816156604425, 0.12967268845190924, 0.5129272490269107, 0.21255470257080908, 1.1147455784158264, 0.6053189380853761, 0.8741876088847811, -1.4708870963608138, 1.1606768236902947, -1.0366226124761504, 0.3298851459593364, 0.25214855299266936, -1.2147494817540572, -0.9692073934014122, -0.3525307421260573, -0.3471350143503415, 1.1915479164686484, -0.3925872816826025, 1.1879526318361548, -1.1853775411981333, -0.7060181566390558, -0.587986421740006, 0.6979326515210571, 1.4402617056379263, 0.2332181300724029, -1.190009135064121, -2.2074236135561542, 0.9239863664814232, -0.6775034976121095, -0.22599005457791116, 0.997034337904372, -1.0576931640556617, -0.8012594157507674, 0.7056806569451592, -0.11480730835197248, 0.7194680807436517, -0.40903518666951144, 0.6586054487581294, 0.8450414437299423, -1.1805986770688168, 1.3147325816738686, 0.6609776351481902, 0.16307449068625007, -1.0381822471835942, -1.003468037064153, -1.7652609151551624, -1.677130569126979, -0.3867436147734942, -0.043921579508683584, -0.4934459403029388, 1.7322992301409257, 0.6179605774237334, 0.1940423403612911, -0.5362767956469552, -0.20857961549520568, -0.8882279400596078, 0.5691596230805598, 0.4319529662324314, 0.307402455363852, 1.8065782534920543, -1.1526628066056452, -1.2642054524499156, 0.17766252983077505, 2.14966167434506, 1.6119903517506196, -0.24727543681478573, -0.17119516239764013, 1.7614867038792519, -0.2352845138269977, 1.091922612926594, -0.3757048866134244, -0.685720063556341, -1.3060733908686195, 1.332460066417033, -0.0899766582335586, 0.09820098412391479, -0.6880800019321492, -0.30749975644401667, -1.5453165392839783, 0.057259317862601085, 0.9233423433311225, -0.32685255475854774, 1.6858883659393311, 0.9880636369517871, -0.2117687500450392, -0.15455697103145047, 1.5669402778063655, 0.7914867925613016, -0.03190317371396772, -0.5892737255336072, 0.030211224661778457, -0.3278219738528652, -1.7821151110947808, 1.5266291310307358, 0.800885503705461, 1.6194129265065136, -0.9368285796063892, -0.1820371021798988, -0.9418242025690663, -1.6717625984019038, -0.7277118482842238, -0.6717464632868874, -0.508018152265877, -1.0423758071902127, -1.0848514755373002, 0.17987530014067593, 1.641639375521459, -0.98319559987103, 0.2512298778477813, -2.1262816570867273, 1.4468223704201069, -0.15025739888103493, -2.2852727301053926, 2.8270688937911754, -0.5102276876467498, -0.08195485649418445, 0.5318374649538267, 0.018923038501235175, -0.12209399155066851, 1.0715591632783221, -1.2904096642262075, -1.9527564623854259, -0.5610765772888551, -0.18480626177318174, -0.541886530968255, -0.256784882601829, 0.8414495916318862, 0.40684501876525564, 1.331445421847139, 0.392089225591716, -0.12247185999625337, -0.8542643789800997, -1.3344013911372374, -0.15313517512943553, 0.6011347069669765, 0.588254946898775, 0.6085829718936638, 1.600745401407988, 2.4820498669842603, 0.4510050524903543, 0.7343140545546403, 0.6357913170423357, 0.11401810667376647, 0.27617754549313794, -0.10123423654880663, -0.10168791828235788, 0.525584178648981, 1.5455579436140867, 1.6140197801054996, 0.35478435311269174, 0.08124265211904454, -0.7106324645585257, -0.15139980749591045, -1.6900833832813464, -0.9098645918633509, -2.2926970762837358, -0.8551442664647901, 0.8446973237404736, 0.7672265759903983, 1.4958043579556781, 0.5313701181646984, 0.1326641061014936, -1.524989888686917, 0.30043617428281644, 0.7916997398256741, -0.7295496482854419, 0.10517885700327075, -0.8526501252561597, 0.38024208308623914, -2.7044616531289845, -0.4186351268026396, -0.38450762831016055, 1.0724201039057075, 0.23459639008138744, 0.13857698145629593, 0.9531603359454035, 0.34505823939104097, -0.8562760550452182, 1.9586851603730024, -0.37982741367962014, -0.8448015492574766, -1.2797005171741311, -1.2069203810870721, 0.233747245722869, -0.6953660489585697, 1.1658844990995432, 1.2835056791350778, -0.28318937405101663, 0.6802539628322593, -0.7224655763379751, -1.5773914603221724, -0.2758178512239434, 0.33285069440112314, -1.0202369786263201, 0.09425008192445976, 0.24165855134686098, 1.048183759282999, -0.3566028729761527, -0.14280813203767798, 0.860916765698592, -0.5554983093228448, -0.9972951303111713, 0.0990771461683793, -0.2936494295683328, 0.5974830830727575, 1.5486616353888552, -0.3733958026208186, 1.5347966820797656, 1.3001375302564964, -1.356852749887053, 0.7780890032738172, -0.6167134091153312, -1.6453431959359295, -1.576154470405435, -0.021942707246093082, 0.4507429731466412, -1.942820192003204, 0.9563441508051181, 0.2160762694863363, 0.10198131938737293, 0.5610793713569487, -0.7192351803429956, -0.26757934785018495, 0.5126354351559333, -0.7691495588160804, 0.6492696741288438, 2.26168304152444, 0.5436200044061363, 0.9230025408782664, -0.38696780362326355, 0.6750653800936846, -1.3002489766237288, -0.13429159740515514, -0.2611819742815461, 0.9004566140276575, 1.5457765779242088, 0.813834233219343, -0.2640296654775034, -0.5023214533836893, -0.5591275332273451, 0.9706307843342499, -1.7488862598436257, 0.45174089413985224, -0.5726013559322352, 0.9435334744399805, 1.7135192780821777, -1.0561911715515226, 0.47602942018039035, 1.8063283719974905, -0.636688530513625, -0.6236715208055557, 0.34685292707224175, 0.08210419898879433, 0.8456531169319162, -0.03604556668275776, 0.4714374607169893, -0.33446825203270225, -0.4746161900768639, 0.41508138422430907, -0.004023716750036164, 1.272426319823319, 0.675066587673383, 0.34579183739352937, 0.09012740158770033, -1.1561800993419424, 0.9696287841030057, -0.946358967998821, -0.627339627756083, 0.5086792922937358, 0.45910678171133945, -0.47576763260556143, 0.7972968793550402, -0.16639010715313196, 1.5842322694980961, 0.7646724960218919, 0.1809057929865927, -2.6040868846995977, -0.8639610211388589, -0.1637291167247112, 0.7559721150094122, 0.8301515818899458, 1.1232614474369793, -0.028971626659527176, -0.5327254724016574, 0.7559148658009005, -0.6803019225122656, -0.05168631889531179, -0.5079274907972391, -1.0304710511141715, 1.5356566174065547, -1.1006092339126758, 0.5564691174988132, -0.671718587733725, 0.29566626554111636, 1.7261221330371033, 1.2459170922309821, -0.36585467719264597, 0.4088842613412259, -0.8185530749317975, -0.9320083431488109, 1.346030369646055, -0.5850377879003854, -0.39786972224814704, 1.225742472069833, 0.2610803244921335, 1.1653155927591463, 0.4096034882305108, 0.2236422043531929, 1.3180066543280158, -1.06300476768897, -1.4223441008072124, 0.7208587843430397, -0.6293811878914242, -0.39218494409286175, -0.7219261228271685, -0.28677636637719817, 0.32408545865208976, -0.5350953356029065, 0.4431160788914869, -0.08475989835718217, -0.5372631417367298, 1.003410119671703, -0.5567327738074517, 0.18411398423699596, 1.0351926578660697, 0.7482284274736988, 0.3846154248893404, 0.29703186026173317, 1.6256393915214618, 0.5417454931171409, 0.9892049192563137, 1.1999915902819422, -0.4437629594939521, 0.6342886876575945, -0.2540686231590402, 0.6635228632701987, -0.0863746816504598, -0.14336801015100561, 0.19855837484419644, 0.5604234675998754, 0.6943092825743786, 0.06863700511874588, -1.1490602461173811, 1.1639889200488547, -0.9504504984161805, -0.6714282236639468, 0.41336034119345194, -0.7731672135339527, 2.322508696033796, -1.6884246548311415, 1.695905414535322, -0.2361265321264915, 0.4199092888708635, -1.3063252417595226, 2.2830087000586907, -1.4350983364180279, -0.3750168776681431, -0.3505576099234748, -0.6902566401491294, -0.7390695239234494, 0.13137392445775606, 1.0094922784043792, -0.16520378506252587, -0.1857495351954293, 0.28882484940296627, -0.5437020564794016, 0.8713462782164673, -0.8836048044024347, 0.5914852359128813, -0.79847291895634, 0.9489006279532309, -0.41567874863960286, 0.37134195344875726, -0.14647242600914323, -0.04377686967188471, -0.28900456282772435, 0.6337809483366149, 0.20270989568064346, 1.8656923501976639, -0.7045304585141084, -0.09399949033189117, -1.4456590783871448, 1.016829792387929, -0.019097019313726284, -0.28528795672389384, 0.48450103437263575, 0.8629140797317346, -0.5941520617684792, -0.8863676191625862, 2.263263240750425, 0.6066521055207299, 0.14704434148003773, 0.8836922850290612, 1.1619013831110767, -0.44892797649201455, -0.6611849593619896, 0.20333787384301447, -0.622617673390457, -0.4247721362925676, 1.2317965913919575, 1.389120917461131, -0.8508850529171128, -0.9048992216905841, 0.3207880199630667, -0.8286457776025455, 1.3247726860662343, 0.6314465163001309, 0.40469772016568706, 0.3479097883804353, -0.7619966184023528, 1.1687311912280933, -1.5307118656437164, 1.6729491491861321, 1.5775768349568968, -0.15665642785199604, 0.3955991547919495, -0.18502811627362, 1.5344993977526267, 0.2910014045629812, 1.645189803317325, -0.10364850469452279, -0.42265602716369843, 1.3593366144144796, 1.2859559468036688, -0.3359893772244806, -2.040680612981941, 0.017586631634430035, -0.7436966664337806, -0.0619255399510405, 1.357461944913464, 2.0255988388130697, 0.6721039054119972, -0.9314738008842361, 1.3778532307536766, -0.24981399790301356, -0.017509858286758745, -0.21023188663121375, 1.2100396773343514, -0.5878566733245616, 0.6411702722225636, -1.334368161598794, 0.4139475590647123, -0.4694860528943416, -1.5501680478362738, 1.2390867122034184, 0.33313497740876385, -1.74873397785183, -0.8925091548797428, 0.3732220760601992, 1.1516948042166284, 0.6517233498239213, 0.3730619545157576, -0.056822270273586314, -1.5576955416387317, -0.12729202682845314, -0.46547662272864126, -1.0159278365937796, 0.24007172969056043, 0.5272066868772172, -1.2227303759040042, 1.5439753261452174, 0.8474671794199751, -0.9897758392298016, -2.091297799163502, 0.6205256709950706, -0.36761253105394354, 0.28854590802181557, -0.7402687760242604, 0.4538176576928001, -0.5076484003227396, -0.7106368727207231, 0.24030229797278574, 0.39038999789746304, -0.5986649729574137, -0.3946658551202863, 0.03579449231973766, 1.094635628670441, 0.352146358050201, 0.3108560015671903, 0.8240150050907582, -0.5467522267114764, -0.4539075335979684, -0.6119368853126542, -0.17589261314349042, 0.5913039085896432, -0.20778860790412465, -1.5801266885829675, 0.6707001457233307, -0.009840947088024875, -0.25273793232855346, 0.6794293096223273, -0.12464988130469545, 0.08121157184253448, -0.5082902614674139, 1.0142765517457453, -0.16899216890150873, 0.8083354854303035, -0.6218168353441347, 0.7347319120133503, 0.8077813599976277, 1.5921923548900156, 0.9335665021290264, -0.8615299293204075, 0.22603530371625824, 1.0529657731876267, -0.06705485107842242, 1.5644910935684206, -0.05298276813697341, 2.385090785721056, -0.09212282315461857, 0.900172896503727, -0.03514212513474264, 0.5134166057556198, 0.4346541833544803, -0.08800646830806147, -1.8702934937449684, -1.651462868620904, -0.05357321033506992, -1.177611616899029, -1.249073137118772, -0.7378083862842354, 0.27893677663591493, 1.5693697420622528, -0.6189244849166535, 0.3470208481618134, -0.4485748145243397, -0.892544070144202, 1.6994777721197332, -0.8960668454720923, -0.84607979146334, 1.716238637275337, 0.1285862399934811, -0.8732987035242843, 0.05942331726041001, -0.3977235292710907, 0.7294797340717084, -0.6053177422704895, 0.6302006549589829, -1.1385983035737302, -1.3904173249386875, -1.329838846955825, -0.2036229613110061, 0.6003258362609504, 1.1535768131267918, -0.37937996365421184, 1.169181860829785, -0.12418921874734155, 0.17383042533052234, -0.481537889868784, 1.3101701230533862, 0.5992340009336387, 1.1060099406888517, -0.2334886726245276, 0.6682622093266681, -0.7883411331092882, 0.9304173484739742, 0.18880191322828202, 0.2429253099894819, 0.32840932863626254, 1.4888090589578324, 0.7441094755022372, 0.6003064183872574, 0.5526677165236636, 1.1142867258749611, -0.7009328587772014, -1.713887972584437, 0.8559483547498408, 1.1096061597040319, -0.5494926083522502, -0.31816516162174957, -0.13177007906232083, 0.6809525941981052, 0.6916920302235213, 0.3229605790161159, 0.7444715135352545, -0.7919799898924343, -0.7939005623300774, -0.8562778403985929, -1.4670632203260634, 1.4937671440848792, 0.9205725482078482, -0.11942750305494426, -1.4979769210339742, -0.79938853710062, -0.3569035621749818, 2.025477795634038, 0.7499604751760377, 0.8101366355130895, 1.8033214988241457, 0.1034380609058387, -1.313994851835685, -0.006203541150737299, 0.00829835412549373, -1.0783155529952624, 1.673370639959754, -0.8759661625718284, -1.3800047698239102, -1.5294043294031003, -0.7395003234699583, -0.07858257559351649, -0.6276895577057493, 0.20293939182017467, 0.7278681556962159, 0.21051006346034823, 1.7092980657094679, -0.07720436434653469, -0.9199228687749581, 0.5557055710656036, 0.012069478067572267, 0.1317102033307551, -0.025195615057342115, 0.8803663402243906, 0.4810513273653718, 2.24285322554312, 1.6213908418199878, 0.641383991684951, -0.4298774311642148, 0.45676015630342487, -1.4547645577971806, 1.4770026709180277, -0.0557137185303508, 0.12985863856283017, -1.4633936923899948, -0.5536046968464772, 0.8968680571798889, 0.22138276600411588, -0.6366897905654244, 0.38616901696479866, 0.8074544218563191, 0.06599525449616717, 0.7365443942986442, -0.4113444784828943, 0.657565746881872, 1.297409506361238, -0.23660472994806803, 0.11826244672242599, -1.0684648715793954, -0.9519689894794527, -0.37290161908427594, 0.7255021325896591, 0.1965241798003828, 1.5844128654463905, -1.9319304998099744, 0.5249490378130609, 0.1547915961890337, -0.9416711476101157, -0.5124682335133077, -0.750663075239628, -1.5088053404855135, -0.06940266056508862, 0.42926680432069175, -1.556598868037818, -0.34792976902450373, -1.914065082298229, 0.7911560005881031, -0.11379349501765466, -0.20807877408454264, -0.1995459917508251, 0.09705925593940228, -0.47003801453826544, -1.9277343433693144, -0.601705653351484, 0.6964140079404187, -0.21800281550701361, -1.3538376283380347, -0.3537785312376512, -1.0715206971685696, 1.5445862655537683, -1.1144282421835123, 0.16305151712678265, -0.8578903056005114, 0.07221209345080441, -1.4784793510691108, -0.09399624770085781, -0.43350579207020906, -1.3253552292412145, -0.20893939950544535, -0.22067355795166374, -0.4947751810335197, -0.98704082390889, -1.152789888389272, -0.17130542206037966, 0.19552845303813196, -1.3445141007819945, 0.7833931540592826, 0.6022295670933976, -0.9149228522872741, -0.2559150513082585, -1.1432972604040466, -0.004885330534659905, 0.7793136671874031, 0.839994047883539, 0.12263513537617234, -0.29803921235121256, -1.7811488933311737, 0.13550452287138529, -0.3622190668517741, 0.6923022251337085, -1.0926634262928177, 0.5902019339441608, -0.5659069280744391, -0.07409007480698994, -0.6062793235044469, -0.0494280249899602, 1.3220206585388907, 0.04940535101478807, -0.08688464354443938, 0.08588557544857985, -0.401469786038712, -0.42096822667708306, 0.24980542963531105, -1.1427112681827725, 1.1283239113918826, -1.1605680103018945, 0.3004461986267594, -1.582820203603348)
xx: IndexedSeq[(Double, Double)] = Vector((2.7695882409509895,1.7180200627538347), (0.14372794474949302,0.7612579495121726), (4.3631104940893835,1.1044351189222115), (-1.3621566952481916,-1.0458887821837508), (3.3360308920367383,0.49496877204701656), (0.43087832803501736,-0.7877659459352377), (0.6234577123367453,2.2839982579960774), (-2.1869356744435975,2.0904918533649792), (2.359998548483437,-0.3154431840702555), (-0.7247639239397248,1.08225194191406), (6.212044356153804,-0.08599324189813665), (-0.8218662613051042,2.038160537801021), (1.1026517251972716,1.129108822212312), (3.0990398913590127,0.0332092866052016), (-1.2319481774525665,-0.8484628252915802), (1.634537917351346,1.7489986908075437), (-0.3402305088068378,0.5905753508188052), (-3.1821232713348735,0.4532853262146371), (3.294828731362106,2.1071390618861483), (3.6630904934954147,-2.4333190509254523), (-2.7098776565047022,1.8666210820194409), (-0.9271615567283469,-0.4716364393080115), (-3.4337942426175596,0.43215718847354717), (-0.1621739404356195,-0.0508145229262295), (-0.5524333003239039,0.2318912449411393), (2.970406774610373,0.6642118433489868), (5.722399747112255,-0.34188724147579436), (5.793033993746275,-0.955949132880485), (5.855544309699007,1.0374062592965703), (0.7298688592112844,0.30231115166753975), (-0.6150366918355494,-0.7053596883786023), (-0.5945299151971124,0.07772257556676218), (0.012460167615885753,0.2734856951677595), (-2.4391045176515496,-1.1171292063497231), (6.5875398123524365,-2.25342400569796), (-2.032021024917923,0.8662107078313865), (0.7749104621938003,-1.2219562085123437), (-0.6296750029126488,-0.049916676934831865), (0.8113279959977424,-0.17692737947420795), (4.00951505753056,0.023142072745985356), (-0.8461222436960973,0.23206737522014134), (4.083353609700269,-0.24576353385861008), (0.6323629679170286,-0.2650095664570966), (-1.5232884547417886,1.0431508512261574), (-1.3956688256785421,1.3426415201672703), (3.2127304146147577,-0.3411574543509265), (2.5190391331302093,-0.05605340120237845), (1.2277258013592707,0.8999076240632354), (0.17212766374877697,2.0209749614041304), (3.090830937353424,-0.6983443417434481), (0.3690777198547921,-0.5973195209029522), (-0.5259374154894529,0.856908070217254), (1.0394651196445837,-1.1828737859192346), (2.6886549656846057,1.510478238660708), (0.38593992465968663,-2.2247519539098066), (0.5743721649166168,-1.5582965573761123), (0.8687114373048672,0.06906490685773707), (0.2666333600121863,-2.042771261983173), (-1.389987712987998,-1.2655530046980348), (0.3644785814447564,0.7776369909128293), (2.684960253921639,0.3805465428130765), (-1.8901082362261161,0.0037395890916070047), (0.6736822480443543,-0.44857098618012636), (-1.2061935146069898,-0.3839517041971513), (6.133402669009728,0.8637075914602533), (1.5843085147781575,1.9008996217205008), (1.4133817442842203,-0.9732951219651634), (4.310016762435671,0.47808650335049846), (3.2703963032515726,0.7810581695915911), (5.512461969036787,1.9073405712038916), (-0.5136133509164615,0.6155484840550374), (1.1273083982162302,1.285720743143111), (-1.3829621674794184,-0.15563461998811107), (1.3343348248052036,1.1963614398365867), (0.7224474404732261,-0.3107393094384793), (0.5536351253903287,0.1442507745437628), (2.046036748345446,-1.0062706124372303), (-1.8689217689355155,-3.2144116143213446), (4.67504915127722,-0.3791151897202505), (4.354907878475408,-1.1018272395002096), (1.0614386884930742,0.014112001386570377), (1.843052755135033,1.417768685115442), (4.0856529781111135,1.849946357990971), (3.547793039794719,2.297256967452079), (4.23806880644579,2.863967849863008), (3.9580194101709942,1.1727349692923696), (1.4428204464300816,0.2638734749999953), (-3.405099154423106,-0.6386146927254167), (2.3306460927731862,-0.7621438861307569), (2.346540691140988,-0.1761694299607887), (3.183641306784483,-1.0034798540546024), (1.3762778036098218,-0.5624664952260037), (-2.165559181699614,0.2742520833159514), (-1.6847306634931067,-1.4493854639080737), (1.7197591122619862,-1.5212897485964094), (-0.09344955678221045,1.4720420607751994), (0.09698661242596862,-0.2291161453735276), (4.505637803764699,-1.7256651550846345), (-0.754783076695082,-0.26044179729408784), (-1.76224461945267,-0.2508065280088541), (3.364919391083711,0.8772202042323987), (1.370296963870416,1.2718923272585114), (-0.5137886460393637,1.2727351580988864), (3.1234888653371864,0.07159473932867604), (1.8900348443202866,-0.05641516601493637), (2.9741034672166675,-1.3063232275399193), (-3.9735036248030013,-1.0188776063643208), (1.9638734156106348,0.20177767101494343), (2.7735622386702277,-0.19326082514830212), (1.5507328328039143,0.46880870506107236), (-0.5448230769210287,0.02644986556162372), (-1.0267496350288878,-1.66251624836091), (-0.35504763114010474,-0.6563531547897851), (2.869361499902716,0.18965199488691234), (-0.9236546043982545,0.8733631401865725), (-1.2175767333417253,-2.624790720525607), (0.763292451609999,0.8857326290925008), (4.033271904943603,1.4816882795472734), (2.1102397085538884,-0.9975488820685854), (1.5013460429633039,0.6481986514102127), (-0.15494393119622218,-0.8749709644202547), (0.12902850393514864,0.7579070844724433), (3.1578635598620353,0.2622465401074379), (-2.0516930271962193,0.6746683513550983), (0.4933462816362998,-1.263448996919519), (2.9696431895602453,-0.7381981835849909), (2.6658616305339335,0.14994282623171254), (1.0019350500021627,0.15669755947388275), (1.5137299840557001,1.520118672383452), (-1.4713055260377117,0.5801060842383697), (-0.16551165755147146,1.0661532398756548), (2.085757774664046,1.9130703293252176), (2.5429532795015692,1.1358559752636856), (3.329103234106629,-0.08681707315553623), (1.9678321015868474,-0.0018107786449524216), (-0.3880290359895091,0.4046562361147712), (-0.3479592898048398,3.3060952703417024), (1.5730475376079784,1.592776279249439), (-0.8938957220891179,0.7656314587366151), (1.395345769184858,-0.9269798956639591), (1.138124708538941,0.39262906647553725), (3.2562732060812123,-0.1179070920628396), (1.0722080654161237,-0.2398427544759008), (-0.7157328602946977,-0.8805376802960007), (-1.9975951468896938,1.185098287008337), (2.032304423284887,2.127758749533745), (4.693608232926701,-1.6193304015036736), (1.531348598652213,0.389753475903186), (-0.45727438450060887,-0.6320665313267828), (1.730561966263675,-0.6071489604350191), (1.9356267639550437,-0.17176036495514585), (-2.589592600218517,2.3082425701198837), (1.6545435662727104,-1.0924630230651688), (3.940612155777473,-0.13377939063602365), (2.4518704454947806,0.19472876609435377), (-0.42715582269583696,-0.6985150412597225), (4.539232311736016,-0.30183923871602275), (3.6556846999080284,0.29017288546487807), (3.1133022715512357,-0.08506161946062703), (1.4515636411226869,0.7880599126205807), (-0.05986081718752101,-0.4152095745664085), (6.544352647719755,-0.5708545974042495), (-2.698401729800919,0.3683149142802798), (-1.707309160150797,0.007561875612265056), (-1.887455554309247,-0.7047584310186249), (-0.031072128941032062,0.8205055007859963), (-0.652975004605133,-0.45649764363696793), (2.3830137054397253,0.14093204594566655), (1.1356861857824312,-0.6566752621490228), (0.8606295364949523,0.6637207009154089), (1.7164236318520327,2.145603957540527), (1.728449111014016,-0.027836928631156135), (0.5834130797061804,0.7423557406071833), (-0.6819133570115252,0.2149356026948634), (-0.7693821492838688,1.2789556634519788), (4.027744219696282,-0.421745799494621), (-0.6019583306146326,-0.624867593419885), (-1.3630635580953854,1.3617530989714166), (1.5966380104586784,0.3786873316449697), (-0.7713061317004588,0.294583494801177), (-0.34187948260528,-0.4244208241183847), (3.2880626472312753,0.37273978804861935), (2.3393598283372663,-2.7888200870508477), (1.8477929806250342,0.13090423295330372), (3.042231436121303,1.4662350241059718), (0.6800512886978949,0.6482747044416928), (0.5464548607419111,0.4099329018255514), (2.5032088667849397,-0.9052966951952968), (3.9874626820119934,-0.7940755000771325), (1.4901753210124256,-0.24743077206220485), (2.052176357069479,0.1052809704647746), (0.6073875189531763,0.7135031861304673), (-2.1642065745709025,0.3890552443985578), (0.3483625004313432,-1.0205843406492123), (2.2794472373542853,0.9951670813851188), (4.77461362780455,0.05125608850433839), (-1.1634226596954593,-0.764826774556313), (3.8433304170724654,0.6698230705476478), (3.074295528275114,-0.6000009790234297), (1.4053840780875588,-0.44656809427334865), (2.234275372775799,0.708775810179257), (4.242978671646227,-0.20972725637246492), (3.4455677370037097,-2.0064825743727734), (1.239206420369047,-0.6445426124434558), (2.514531088837198,-1.1342463969194072), (1.4951333226802432,-0.754557781468656), (5.367141330361266,1.486456873017418), (-0.8039923086153469,0.07209103107719902), (0.4203104910740978,-1.6002365728809387), (-2.285673176809458,-1.046111170467302), (2.1516097822838653,1.5026911880047211), (0.8172145721891397,-0.1457843644663685), (1.8322422907585296,-0.13947748839418284), (3.1842895245518608,0.42851687041940495), (-1.7547571482467195,-0.3451690864154465), (-1.538753163209698,-0.982541496031045), (-1.3317188332097145,1.0986738175653772), (4.8114219888457335,0.6994670164774653), (0.6645477439321279,1.4268673094155087), (-0.9584054649931435,1.1377152102285677), (-0.9121825903494123,1.681158675564279), (0.43158121119227055,-1.0243192501461797), (-1.3202628961334066,0.7157476281459816), (1.1420397909675593,0.1987225329603131), (2.5750492118414954,-0.8995597231470148), (0.6498283686529067,-0.6454347709276017), (-0.46822705319168634,-1.210748247776176), (-0.9151975437595088,-1.2029648168386262), (0.1254871446823136,-0.5850933305769891), (0.12116056117790508,0.20655562292795113), (-1.1449845637585945,0.19367807044160337), (0.4950229149589839,-1.9225118432641704), (-1.7438838241725643,-0.3891688523373755), (2.828814090781172,0.750778001298227), (-0.2165784557889936,-1.4563569125990905), (-3.343337774916857,0.32178951363534475), (2.6965964356635457,-0.42696414784080483), (-0.22563171379209623,-0.5745967327557552), (2.169793350754831,-0.33987806460763226), (0.16039716195181464,-1.2201444717297956), (2.4055739064960906,-0.4804118001724722), (-0.4084575409056723,0.08573017271990857), (-0.3007511526240896,1.2739693849684661), (1.4188439249260876,-0.6860805257029206), (0.7810300471920533,-0.12149128270027325), (2.192372205732967,-0.13213792842778785), (0.5971542123004031,2.1931880748351817), (3.1849107981978415,-1.4266518171274647), (-2.7412457708912057,-1.2862151522144258), (0.07721968922783717,-1.3894824574835245), (0.3867325803769627,0.42483399623760804), (-1.654986538485435,1.8191505956546687), (-3.304801892698313,1.3970685728640833), (3.5304781805183216,0.9731809955429377), (0.09908407575295919,0.5893405158842995), (-0.2949003636490535,-0.3828769935559573), (0.8888645292993355,1.0564581904685673), (1.927377777716643,1.2069544831918129), (3.8265517265141678,-0.35225447445437746), (1.0166695905676997,0.980075180240185), (0.4185108078771228,-0.2214099969048667), (-0.005985818234146345,-0.7765925304274375), (1.922293143142256,-0.40713952940074355), (2.123496400516054,-0.3220702027858841), (1.0139763350056772,-0.7156418098061273), (0.7102937301007382,-0.8797952571244195), (-0.1478242221471764,0.711322217928678), (0.47357995320680935,-1.4874834705020474), (3.9864202379123608,0.26569947209899103), (-1.1041765309750793,-0.6733470729032577), (1.4321706851148899,0.3447471414040174), (-1.4232782205909342,1.2318617256455997), (5.539141875796626,1.44689615341149), (3.8601415354756483,0.8105317859479295), (-0.6288280510015629,1.558192810324353), (1.6008109497181122,1.296473628465781), (-0.6765547301711912,-0.3827008454322967), (-1.0397151933627296,-0.8097542023433437), (0.43276204279473673,1.5251085260269759), (2.460457960365949,0.6212940140441159), (-2.569829436532503,0.5407362869930761), (2.0332909854405594,0.41821399753599575), (-1.5698881042429633,-0.8079238389605706), (-0.9383862850909983,-0.9085463275200037), (2.254971850143108,-0.6225816156604425), (1.8641503982190413,0.12967268845190924), (-2.0135007028968093,0.5129272490269107), (-0.6984326605577915,0.21255470257080908), (-4.26129436701007,1.1147455784158264), (-0.7451757933596983,0.6053189380853761), (-0.07028112001632136,0.8741876088847811), (-1.5174034084054537,-1.4708870963608138), (-1.6107919853671397,1.1606768236902947), (-0.8274055233842161,-1.0366226124761504), (-0.3303071592477751,0.3298851459593364), (3.1319166054508423,0.25214855299266936), (1.424527958434939,-1.2147494817540572), (-0.11986174677502315,-0.9692073934014122), (2.541412494185315,-0.3525307421260573), (0.4269336666911082,-0.3471350143503415), (-1.7733114534363468,1.1915479164686484), (3.2104657513310255,-0.3925872816826025), (0.4250902712636889,1.1879526318361548), (1.525159011473101,-1.1853775411981333), (-0.11970272504187296,-0.7060181566390558), (-2.0907832634272294,-0.587986421740006), (1.9951008927962628,0.6979326515210571), (0.6850835688193767,1.4402617056379263), (-2.087495290128881,0.2332181300724029), (-0.6319303959679023,-1.190009135064121), (0.11630598344989707,-2.2074236135561542), (2.966537816222707,0.9239863664814232), (4.250869748158925,-0.6775034976121095), (-0.613438066703744,-0.22599005457791116), (-0.21636936436956855,0.997034337904372), (1.954859806581149,-1.0576931640556617), (-3.331153046761135,-0.8012594157507674), (4.3313518476524795,0.7056806569451592), (1.6666279337294463,-0.11480730835197248), (2.121390428920673,0.7194680807436517), (0.04231190375815563,-0.40903518666951144), (0.4059877273175234,0.6586054487581294), (2.8995924977716703,0.8450414437299423), (-1.2253677492365114,-1.1805986770688168), (-2.7427962218054307,1.3147325816738686), (-0.813618802213371,0.6609776351481902), (-2.463342947018564,0.16307449068625007), (-0.11316185130832124,-1.0381822471835942), (-0.9793737992320104,-1.003468037064153), (-2.043879492922081,-1.7652609151551624), (-1.8016205521671549,-1.677130569126979), (0.6606815251257572,-0.3867436147734942), (1.9379840701619906,-0.043921579508683584), (-0.5552026978564197,-0.4934459403029388), (2.4998728500757776,1.7322992301409257), (0.49333855243636493,0.6179605774237334), (-1.73167695080304,0.1940423403612911), (-1.3509950907984467,-0.5362767956469552), (-1.7807470585923681,-0.20857961549520568), (-2.509045772341776,-0.8882279400596078), (0.9773125230549433,0.5691596230805598), (0.8657847019995536,0.4319529662324314), (1.4428603935729856,0.307402455363852), (5.206273578636806,1.8065782534920543), (3.5644064837330953,-1.1526628066056452), (-1.3907966040416313,-1.2642054524499156), (3.278891341161479,0.17766252983077505), (0.4136696528217093,2.14966167434506), (3.6647950629078725,1.6119903517506196), (0.2349585327216671,-0.24727543681478573), (1.7965992385416258,-0.17119516239764013), (0.3257535015012287,1.7614867038792519), (-0.6243861489768252,-0.2352845138269977), (1.8344741743983246,1.091922612926594), (2.705956122226601,-0.3757048866134244), (2.266989875313474,-0.685720063556341), (6.017366534311652,-1.3060733908686195), (1.61793022523292,1.332460066417033), (3.409835424815995,-0.0899766582335586), (2.0607129627533256,0.09820098412391479), (-0.0569293203710064,-0.6880800019321492), (3.0519006385051917,-0.30749975644401667), (1.1659401372201552,-1.5453165392839783), (1.6459523637820013,0.057259317862601085), (2.4633065223801913,0.9233423433311225), (1.6480473378492224,-0.32685255475854774), (1.8000222018004628,1.6858883659393311), (2.9019242923278807,0.9880636369517871), (1.828333160098154,-0.2117687500450392), (0.1825471942073068,-0.15455697103145047), (1.8393614379755125,1.5669402778063655), (-0.6386083651659122,0.7914867925613016), (1.3334722495285987,-0.03190317371396772), (-0.1004824180635544,-0.5892737255336072), (1.28776175649449,0.030211224661778457), (5.332720323285932,-0.3278219738528652), (-0.35729715603488543,-1.7821151110947808), (1.284601603301558,1.5266291310307358), (1.5863513393343682,0.800885503705461), (1.5717255822207639,1.6194129265065136), (1.9266846823962425,-0.9368285796063892), (1.2152209764315616,-0.1820371021798988), (2.839720426328611,-0.9418242025690663), (-1.5700149097490268,-1.6717625984019038), (1.4897564971545172,-0.7277118482842238), (3.0853187556604977,-0.6717464632868874), (-0.22619785161755468,-0.508018152265877), (-0.0746729644432349,-1.0423758071902127), (5.216079036303841,-1.0848514755373002), (2.3430079371131263,0.17987530014067593), (-0.6098606117170651,1.641639375521459), (3.5068950097265916,-0.98319559987103), (3.070058268862877,0.2512298778477813), (-1.6225068083839989,-2.1262816570867273), (-4.891266045456622,1.4468223704201069), (-1.2182776210406074,-0.15025739888103493), (2.290554213031988,-2.2852727301053926), (1.0200981571058119,2.8270688937911754), (-2.268307590933392,-0.5102276876467498), (-0.10551156398944239,-0.08195485649418445), (0.4467947728101834,0.5318374649538267), (-0.7932420041634938,0.018923038501235175), (0.9142994546502001,-0.12209399155066851), (5.743485509405601,1.0715591632783221), (1.451017825367392,-1.2904096642262075), (0.4096378796222342,-1.9527564623854259), (1.96881934793001,-0.5610765772888551), (-1.0972657720507195,-0.18480626177318174), (-0.9735725731054337,-0.541886530968255), (0.8476606154778268,-0.256784882601829), (2.1582186853736736,0.8414495916318862), (4.55364368265798,0.40684501876525564), (-0.686279626729805,1.331445421847139), (2.5555860710874225,0.392089225591716), (2.7331356843168306,-0.12247185999625337), (4.518435073403177,-0.8542643789800997), (2.2854598570301254,-1.3344013911372374), (-2.0870424177069693,-0.15313517512943553), (1.4978678548929367,0.6011347069669765), (-0.8976049945840789,0.588254946898775), (3.283519638649413,0.6085829718936638), (1.728791081219052,1.600745401407988), (1.9800506661907966,2.4820498669842603), (0.9898103821529822,0.4510050524903543), (-2.494335842642964,0.7343140545546403), (3.7561664386643825,0.6357913170423357), (5.1132488916961885,0.11401810667376647), (-2.0737802343753646,0.27617754549313794), (4.355248811144406,-0.10123423654880663), (-0.32156149095706077,-0.10168791828235788), (2.5049198139799906,0.525584178648981), (0.31959628776324056,1.5455579436140867), (2.4410000582140103,1.6140197801054996), (2.9989255370999355,0.35478435311269174), (-2.795214850339636,0.08124265211904454), (0.33970118219286105,-0.7106324645585257), (0.5439306211765949,-0.15139980749591045), (-1.8548679424460555,-1.6900833832813464), (1.3384936198148278,-0.9098645918633509), (2.657170418993738,-2.2926970762837358), (0.4813826405683277,-0.8551442664647901), (-0.3914733278423883,0.8446973237404736), (-0.8565187795935223,0.7672265759903983), (1.0803503437087876,1.4958043579556781), (-1.3462119220312303,0.5313701181646984), (5.682532851517794,0.1326641061014936), (1.1999086980404865,-1.524989888686917), (2.4216105976403632,0.30043617428281644), (3.1107976470494987,0.7916997398256741), (0.09683232609369341,-0.7295496482854419), (0.8850181243597492,0.10517885700327075), (4.080505409548124,-0.8526501252561597), (2.882874607902859,0.38024208308623914), (-0.3377855595926509,-2.7044616531289845), (1.2258031860831107,-0.4186351268026396), (1.0882302551123175,-0.38450762831016055), (-3.8945830274669406,1.0724201039057075), (0.8110469390963716,0.23459639008138744), (-0.9574596627970309,0.13857698145629593), (1.117847072300214,0.9531603359454035), (5.82462356700895,0.34505823939104097), (0.3033508459697294,-0.8562760550452182), (-1.1438179620554507,1.9586851603730024), (2.842376390102137,-0.37982741367962014), (5.077682920984305,-0.8448015492574766), (0.16500893412531437,-1.2797005171741311), (-1.2603514467718813,-1.2069203810870721), (-1.2055201602764183,0.233747245722869), (1.7584924763911771,-0.6953660489585697), (0.4755174281786603,1.1658844990995432), (1.3823525034011217,1.2835056791350778), (-0.25550936571595195,-0.28318937405101663), (2.371102278113568,0.6802539628322593), (2.0857062916769395,-0.7224655763379751), (-1.3860084193241287,-1.5773914603221724), (4.133847224109132,-0.2758178512239434), (1.9921945568407637,0.33285069440112314), (1.170516756536207,-1.0202369786263201), (2.494329598135376,0.09425008192445976), (2.9323878881707026,0.24165855134686098), (2.4275962612545294,1.048183759282999), (-1.685210476848662,-0.3566028729761527), (-5.50894499711671,-0.14280813203767798), (0.7555934739074583,0.860916765698592), (1.5615955966912691,-0.5554983093228448), (0.6424964473160483,-0.9972951303111713), (0.5578325343364791,0.0990771461683793), (1.7633152144535225,-0.2936494295683328), (-0.5486625382561401,0.5974830830727575), (3.711638649378193,1.5486616353888552), (1.7219434815425188,-0.3733958026208186), (0.21347316918544967,1.5347966820797656), (-0.2381607138285975,1.3001375302564964), (-0.4943225686323034,-1.356852749887053), (1.3233181663637466,0.7780890032738172), (2.874542627271738,-0.6167134091153312), (-1.2884041997444475,-1.6453431959359295), (-0.6823271224263037,-1.576154470405435), (4.164273176362354,-0.021942707246093082), (-0.5666554259914869,0.4507429731466412), (0.5960472751856558,-1.942820192003204), (1.987339228014005,0.9563441508051181), (-0.42148925080930133,0.2160762694863363), (3.57625745119623,0.10198131938737293), (1.403149411102508,0.5610793713569487), (2.270337735904603,-0.7192351803429956), (-1.58931221234994,-0.26757934785018495), (0.13128349748505797,0.5126354351559333), (1.798805640548819,-0.7691495588160804), (-2.3387784402205525,0.6492696741288438), (-3.1873878731460747,2.26168304152444), (3.37723378032188,0.5436200044061363), (0.6655556582577118,0.9230025408782664), (0.5920391865025911,-0.38696780362326355), (-2.0738648990409914,0.6750653800936846), (1.9535655762688429,-1.3002489766237288), (2.120654894307195,-0.13429159740515514), (1.7548919635039737,-0.2611819742815461), (3.606959388968125,0.9004566140276575), (1.9058077963362696,1.5457765779242088), (-0.277735532923846,0.813834233219343), (1.7932735141092266,-0.2640296654775034), (2.115318771475017,-0.5023214533836893), (1.8966456148008137,-0.5591275332273451), (0.41981909504539694,0.9706307843342499), (-0.7900089053488428,-1.7488862598436257), (0.6088361727587679,0.45174089413985224), (0.5139144525003939,-0.5726013559322352), (0.2882137314099307,0.9435334744399805), (2.1497553499564983,1.7135192780821777), (-0.6604142120376058,-1.0561911715515226), (3.99097440528307,0.47602942018039035), (5.832632577473981,1.8063283719974905), (7.1001932424406995,-0.636688530513625), (0.5369780930552249,-0.6236715208055557), (0.5986221682643068,0.34685292707224175), (-2.582516235989321,0.08210419898879433), (3.0931648516158123,0.8456531169319162), (2.2617305609606038,-0.03604556668275776), (2.4541231800578944,0.4714374607169893), (1.4670390377209404,-0.33446825203270225), (4.763302702427172,-0.4746161900768639), (1.4541179259336412,0.41508138422430907), (1.5431868219751557,-0.004023716750036164), (0.12507017486343897,1.272426319823319), (1.1647673167502066,0.675066587673383), (0.04242287352888663,0.34579183739352937), (0.09825850851519813,0.09012740158770033), (1.6853280367313568,-1.1561800993419424), (0.09159896416334778,0.9696287841030057), (-0.9587837141256221,-0.946358967998821), (-0.6616282429966662,-0.627339627756083), (1.0567570700051607,0.5086792922937358), (3.799863072036805,0.45910678171133945), (1.3494683860193444,-0.47576763260556143), (1.0542968271524977,0.7972968793550402), (0.17989559132210153,-0.16639010715313196), (1.2013846428309973,1.5842322694980961), (0.8206872887709004,0.7646724960218919), (3.7562699229401053,0.1809057929865927), (1.4587649934031344,-2.6040868846995977), (-2.2359873367940906,-0.8639610211388589), (2.375175559350293,-0.1637291167247112), (0.7316886300922143,0.7559721150094122), (0.0042779798365065735,0.8301515818899458), (3.198678271123575,1.1232614474369793), (1.6968211714345809,-0.028971626659527176), (-0.5386737859923785,-0.5327254724016574), (-1.030053275945225,0.7559148658009005), (1.3708053913361908,-0.6803019225122656), (5.4519275006444925,-0.05168631889531179), (1.8604883242500105,-0.5079274907972391), (5.4373716919386474,-1.0304710511141715), (1.5261535842714302,1.5356566174065547), (-1.0089850272946626,-1.1006092339126758), (-2.4732194549312285,0.5564691174988132), (1.1603118294770915,-0.671718587733725), (1.7334258899982056,0.29566626554111636), (1.4664725495285897,1.7261221330371033), (2.1131963541725005,1.2459170922309821), (-0.26669042028663603,-0.36585467719264597), (4.503951271053032,0.4088842613412259), (3.4311976307926226,-0.8185530749317975), (-0.3917734687622483,-0.9320083431488109), (-0.30017748727872573,1.346030369646055), (-0.18532621794604043,-0.5850377879003854), (3.694905009226925,-0.39786972224814704), (-0.7150227409266814,1.225742472069833), (1.4252098537389557,0.2610803244921335), (3.8930287891732007,1.1653155927591463), (2.2965882771113217,0.4096034882305108), (0.5083148411982632,0.2236422043531929), (2.1235698631548052,1.3180066543280158), (1.5552501716478493,-1.06300476768897), (1.9681346986160557,-1.4223441008072124), (3.8624128273167893,0.7208587843430397), (3.6909810968889225,-0.6293811878914242), (-1.1180046805446877,-0.39218494409286175), (3.739590423731827,-0.7219261228271685), (-0.13650169785941513,-0.28677636637719817), (3.0758433786474533,0.32408545865208976), (0.6830155383428913,-0.5350953356029065), (0.4736024062176004,0.4431160788914869), (-0.10706976582033167,-0.08475989835718217), (-0.4316760804770359,-0.5372631417367298), (2.7777385963695047,1.003410119671703), (2.48233930467502,-0.5567327738074517), (-1.1602476904399737,0.18411398423699596), (2.3908151155321002,1.0351926578660697), (-0.7975771941263998,0.7482284274736988), (2.6643754068283503,0.3846154248893404), (0.1480311284936099,0.29703186026173317), (0.23105680234390946,1.6256393915214618), (3.587027074607367,0.5417454931171409), (0.6486465295637008,0.9892049192563137), (2.7160651812122323,1.1999915902819422), (-0.6472665087766578,-0.4437629594939521), (1.2757039297324804,0.6342886876575945), (1.1068122550427113,-0.2540686231590402), (-2.361879336991805,0.6635228632701987), (4.518372644338293,-0.0863746816504598), (4.137946086059783,-0.14336801015100561), (3.6129923338454564,0.19855837484419644), (-1.9283362870442553,0.5604234675998754), (2.1263802863804537,0.6943092825743786), (1.0188713827793372,0.06863700511874588), (0.7856247958249603,-1.1490602461173811), (1.4077406059914197,1.1639889200488547), (-0.15775259387789808,-0.9504504984161805), (0.15468502073756762,-0.6714282236639468), (-3.4240713294502303,0.41336034119345194), (-3.5660567652937596,-0.7731672135339527), (-1.4604462603697566,2.322508696033796), (1.1403044401959828,-1.6884246548311415), (0.3780730728707078,1.695905414535322), (0.08422739281580782,-0.2361265321264915), (1.2032502068747215,0.4199092888708635), (1.7011541294300803,-1.3063252417595226), (-0.16569612015490365,2.2830087000586907), (-0.429033505337042,-1.4350983364180279), (1.8107480066735713,-0.3750168776681431), (-0.9444884495712824,-0.3505576099234748), (-0.2513340470946843,-0.6902566401491294), (1.4265081499336545,-0.7390695239234494), (-0.6252397207210769,0.13137392445775606), (4.578381673274146,1.0094922784043792), (0.3583027385145291,-0.16520378506252587), (2.6079421210893434,-0.1857495351954293), (0.974872763967577,0.28882484940296627), (-1.1823119184850772,-0.5437020564794016), (3.158174583733408,0.8713462782164673), (1.9345779235576788,-0.8836048044024347), (0.03133454553210202,0.5914852359128813), (-2.635292478103996,-0.79847291895634), (1.595417079526967,0.9489006279532309), (-3.6708698986179247,-0.41567874863960286), (0.6526779852104553,0.37134195344875726), (2.5389686555005255,-0.14647242600914323), (1.4820557578232854,-0.04377686967188471), (1.3341844355695307,-0.28900456282772435), (-0.4507410311150657,0.6337809483366149), (0.4783948902585634,0.20270989568064346), (0.029542378208074926,1.8656923501976639), (-1.0057807295345391,-0.7045304585141084), (2.7389192340324473,-0.09399949033189117), (0.012276931409287206,-1.4456590783871448), (2.4886654639844155,1.016829792387929), (4.25718984365763,-0.019097019313726284), (0.06086552781930765,-0.28528795672389384), (-2.0266258174224,0.48450103437263575), (-2.3769441249429777,0.8629140797317346), (-1.8423137501054927,-0.5941520617684792), (0.06068912362447898,-0.8863676191625862), (3.729624450684784,2.263263240750425), (1.4042829893379563,0.6066521055207299), (-0.6420338301542439,0.14704434148003773), (-0.30470796074859297,0.8836922850290612), (0.4107302154423097,1.1619013831110767), (1.2979134254705207,-0.44892797649201455), (2.1068984589359654,-0.6611849593619896), (-1.4032333201155343,0.20333787384301447), (2.5689825835070406,-0.622617673390457), (-3.9042461972777067,-0.4247721362925676), (-2.1372144270999685,1.2317965913919575), (1.0959308632084017,1.389120917461131), (0.29760378693488043,-0.8508850529171128), (-2.126310140538716,-0.9048992216905841), (-1.6155274652631295,0.3207880199630667), (2.824940192165557,-0.8286457776025455), (1.6789730105330256,1.3247726860662343), (-1.2997965586855882,0.6314465163001309), (-0.4781315920853715,0.40469772016568706), (-1.0989098065345342,0.3479097883804353), (-0.3758940510344948,-0.7619966184023528), (-1.2687734253210827,1.1687311912280933), (0.055025992085897646,-1.5307118656437164), (0.6764808088548009,1.6729491491861321), (0.11345990754926616,1.5775768349568968), (2.4912055171988583,-0.15665642785199604), (1.959929651335706,0.3955991547919495), (1.8429640429038348,-0.18502811627362), (2.5787278374863964,1....

val rddLR = sc.parallelize(yx)
rddLR.take(5)

rddLR: org.apache.spark.rdd.RDD[(Double, Double, Double)] = ParallelCollectionRDD[3409] at parallelize at <console>:47
res8: Array[(Double, Double, Double)] = Array((10.203590769567299,2.7695882409509895,1.7180200627538347), (1.00829352268442,0.14372794474949302,0.7612579495121726), (10.786370777983102,4.3631104940893835,1.1044351189222115), (-3.1732368758972376,-1.3621566952481916,-1.0458887821837508), (8.730144384554382,3.3360308920367383,0.49496877204701656))

val dfLR = rddLR.toDF("y","x1","x2")
dfLR.show
dfLR.show(5)

+-------------------+-------------------+--------------------+
|                  y|                 x1|                  x2|
+-------------------+-------------------+--------------------+
| 10.203590769567299| 2.7695882409509895|  1.7180200627538347|
|   1.00829352268442|0.14372794474949302|  0.7612579495121726|
| 10.786370777983102| 4.3631104940893835|  1.1044351189222115|
|-3.1732368758972376|-1.3621566952481916| -1.0458887821837508|
|  8.730144384554382| 3.3360308920367383| 0.49496877204701656|
| 3.6306482079456517|0.43087832803501736| -0.7877659459352377|
|  4.907966585602094| 0.6234577123367453|  2.2839982579960774|
|-1.3025719171703307|-2.1869356744435975|  2.0904918533649792|
|  6.866246142975089|  2.359998548483437| -0.3154431840702555|
| 0.9608351137680712|-0.7247639239397248|    1.08225194191406|
| 13.600947763365298|  6.212044356153804|-0.08599324189813665|
| 2.9854820863814098|-0.8218662613051042|   2.038160537801021|
|  4.375902013814805| 1.1026517251972716|   1.129108822212312|
|  6.912624727047396| 3.0990398913590127|  0.0332092866052016|
|-1.5065419073865634|-1.2319481774525665| -0.8484628252915802|
|  6.105408700939455|  1.634537917351346|  1.7489986908075437|
| -0.689483950339185|-0.3402305088068378|  0.5905753508188052|
| -5.389879584585515|-3.1821232713348735|  0.4532853262146371|
|  9.043787447485506|  3.294828731362106|  2.1071390618861483|
|  6.744261020266554| 3.6630904934954147| -2.4333190509254523|
+-------------------+-------------------+--------------------+
only showing top 20 rows

+-------------------+-------------------+-------------------+
|                  y|                 x1|                 x2|
+-------------------+-------------------+-------------------+
| 10.203590769567299| 2.7695882409509895| 1.7180200627538347|
|   1.00829352268442|0.14372794474949302| 0.7612579495121726|
| 10.786370777983102| 4.3631104940893835| 1.1044351189222115|
|-3.1732368758972376|-1.3621566952481916|-1.0458887821837508|
|  8.730144384554382| 3.3360308920367383|0.49496877204701656|
+-------------------+-------------------+-------------------+
only showing top 5 rows

dfLR: org.apache.spark.sql.DataFrame = [y: double, x1: double ... 1 more field]

// importing for regression
import org.apache.spark.ml.regression.LinearRegression
import org.apache.spark.ml.linalg._

val lm = new LinearRegression
lm.explainParams
lm.getStandardization
lm.setStandardization(false)
lm.getStandardization
lm.explainParams

import org.apache.spark.ml.regression.LinearRegression
import org.apache.spark.ml.linalg._
lm: org.apache.spark.ml.regression.LinearRegression = linReg_2e40e3298ad4
res10: String =
aggregationDepth: suggested depth for treeAggregate (>= 2) (default: 2)
elasticNetParam: the ElasticNet mixing parameter, in range [0, 1]. For alpha = 0, the penalty is an L2 penalty. For alpha = 1, it is an L1 penalty (default: 0.0)
featuresCol: features column name (default: features)
fitIntercept: whether to fit an intercept term (default: true)
labelCol: label column name (default: label)
maxIter: maximum number of iterations (>= 0) (default: 100)
predictionCol: prediction column name (default: prediction)
regParam: regularization parameter (>= 0) (default: 0.0)
solver: the solver algorithm for optimization. If this is not set or empty, default value is 'auto' (default: auto)
standardization: whether to standardize the training features before fitting the model (default: true, current: false)
tol: the convergence tolerance for iterative algorithms (>= 0) (default: 1.0E-6)
weightCol: weight column name. If this is not set or empty, we treat all instance weights as 1.0 (undefined)

// Transform data frame to required format
val dflr = (dfLR map {row => (row.getDouble(0), 
           Vectors.dense(row.getDouble(1),row.getDouble(2)))}).
           toDF("label","features")
dflr.show(5)

+-------------------+--------------------+
|              label|            features|
+-------------------+--------------------+
| 10.203590769567299|[2.76958824095098...|
|   1.00829352268442|[0.14372794474949...|
| 10.786370777983102|[4.36311049408938...|
|-3.1732368758972376|[-1.3621566952481...|
|  8.730144384554382|[3.33603089203673...|
+-------------------+--------------------+
only showing top 5 rows

dflr: org.apache.spark.sql.DataFrame = [label: double, features: vector]

// Fit model
val fit = lm.fit(dflr)
fit.intercept

fit: org.apache.spark.ml.regression.LinearRegressionModel = linReg_2e40e3298ad4
res15: Double = 1.5048401230834922

fit.coefficients

res16: org.apache.spark.ml.linalg.Vector = [2.000334475296524,0.9990998828588862]

val summ = fit.summary

summ: org.apache.spark.ml.regression.LinearRegressionTrainingSummary = org.apache.spark.ml.regression.LinearRegressionTrainingSummary@3a691a44

summ.r2

res18: Double = 0.9440325886959734

summ.rootMeanSquaredError
summ.coefficientStandardErrors
summ.pValues
summ.tValues
summ.predictions
summ.residuals

This gives you more on doing generalised linear modelling in Scala. But let's go back to out pipeline using the power-plant data.

<p> <a href="http://spark.apache.org/docs/latest/index.html" target="_blank"> Fallback link for browsers that, unlikely, don't support frames </a> </p>