Tag: regression

Feed: SAS Blogs. Author: Sophia Rowland. Before joining SAS, I was unaware of the possibilities around hyperparameter autotuning. When trying to maximize model performance, I would code up cumbersome pipelines trying multiple hyperparameter values across multiple models. This effort created a better performing model, but it still took up a large chunk of my time. In SAS VDMML, adding hyperparameter autotuning only requires an extra line of code or a button click. While SAS searches the hyperparameter space in the background, I am free to pursue other work. Hyperparameter autotuning is now one of my favorite machine learning capabilities (right ... Read More

Feed: SAS Blogs. Author: SAS Taiwan. 決策樹結果重在商業邏輯的解釋 決策樹不似其他預測模型方法，依照每一觀察者的不同的特徵屬性值，計算出各自的發生機率值，它則是某一區間內的特徵屬性給定一個分數，歸屬在同一條預測規則下的觀測點，都會給定同一個值，發生機率都會判定相同，所以會有一定的誤差值。也就造成決策樹的準確度雖然不是最佳，但由於其容易表達、方便解讀，對於入門data mining的初階miner，賽博士卻是大大推薦決策樹一定要好好學習。模型的準確度在初學時，常會被我們認為是分析結果的重點。事實上，模型結果具有商業解釋性才更勝於一切，模型看似準確高，但產出的模型結果卻是分析人員或產業專家無法解釋，其實就等於是無用的模型。而模型準確度也可透過資料的處理或重要衍生變數的投入，提升模型的準度。準度也必需要禁得起時間的考驗，也就是每定期的模型驗證，來確定模型的穩定性。過度配適(model over-fitting)也可能造成一時的模型預測正確率很高，但幾期的時間觀察下來，卻發現模型越來越失準。(模型評估的方法，我們可以另闢一期來特別說明)。 SAS EM決策樹混合各種決策樹演算方法 我們常聽到的決策樹演算方法有：ID3、CHID、CART、C4.5、C5等方法，SAS EM的決策樹模型分析方法則混合了上述所提各種的決策樹演算法，在單一決策樹分析節點裡就可以連結各種決策樹處理演算法，分析者只需將欲輸入的資料集的資料型態設定好，在SAS EM決策樹分析節點的分割規則(Splitting Rule)的各項參數，進行適當調整，即可選擇想要的演算方法進行預測模型建置。 表一、決策樹演算法之比較 資料屬性 分割規則 修剪樹規則 Quinlan（1979） 離散型資料 Entropy、 Gain Ratio Predicted Error Rate Quinlan（1993） 離散型資料 Gain Ratio Predicted Error Rate Kass（1980） 離散型資料 Chi-Square Test No Pruning Briemen（1984） 連續型資料 Gini Index Entire Error Rate 運用SAS EM決策樹產生CART分析 CART (Classification and Regression Trees)演算方法是以每個節點的動態臨界值作為條件判斷式，藉由單一輸入的變數函數，在每個節點分隔資料，並建立一個二元決策樹，CART是使用 Gini Ratio來衡量指標，如果分散的指標程度很高，表示資料中分佈許多類別，相反的，如果指標程度越低，則代表單一類別的成員居多。C4.5 與CART 建構過程非常類似，兩者之間最大不同就在於節點分支的數目，C4.5 會在每一個節點產生不同數目分支，分支數目會決定於行為變項中類別的個數。而CART 只會在每一個節點產生兩個分支。C4.5 是較新出現的決策樹演算法，它的前身即為ID3。 在SAS EM決策樹裡產生CART分析，必須進行以下參數調整： 運用SAS EM決策樹產生CHAID分析  CHAID (Chi-Square Automatic Interaction Detector)演算法為利用卡方分析(Chi-Square Test)預測二個變數是否需要合併，如能夠產生最大的類別差異的預測變數，將成為節點的分隔變數。透過計算節點中類別的 P值 (P-Value)，以P值大小來決定決策樹是否繼續生長，所以不需像C4.5或CART要再做決策樹修剪的動作。CHAID 與CART、C4.5 之差異在於，CHAID只限於處理類別變數，如連續變數必須採用區段的方式，轉換成類別變數。另一差異部分在於修剪的部分，CART、C4.5 是先過度套用資料訓練，之後再修剪。但CHAID 則是在過度套用之前即停止支點蔓生擴大。 ... Read More

Feed: SAS Blogs. Author: SAS Taiwan. 擬定分析主題後，在正式進入分析之前，分析材料的準備是資料探勘中極重要的工作環節，分析者須針對欲研究主題列示一份的變數清單，應依分析目的與假設、研究對象與範圍等列式出研究主題可能的相關變數。 在大型的分析專案，初階資料準備過程，容易產出動輒上百個分析者認為可能相關的變數，這些變數有些可能重要，也可能大部分都無相關，需要進一步一一與目標變數進行分析與檢定，才能確認變數的重要性，描述性統計分析或透過圖形檢視的資料瀏覽程序是一種方法，但對於大量分析變數投入時，則會相對耗時。 針對大量資料的變數篩選，EM則提供 Variable Selection node 快速的鑑別哪些變數對預測模型是具有解釋能力的，協助進行變數化簡的資料處理工作，經過 Variable Selection node 的變數檢定程序，與目標變數無相關、相關性小或變數解釋能力小，變數角色會被設為拒絕(Reject)，被設為拒絕的變數仍會繼續帶入後續節點，但是不會帶入預測分析裡進行模型建置。 Variable Selection node 透過統計檢定的方法來快速找出解釋變數與目標變數間的關係，提供模型變數預篩選功能。 同時可用以進行分析假設的驗證，確認真實資料是否與分析假設相符，若與假設相符，則可進一步得知其重要程度，反之，可能確認該變數與目標變數無關。 但實務上有時我們會對假設認為應該顯著但分析結果卻拒絕的變數，重新進行資料檢視或進行適當的資料處理來改善它的模型解勢力，特別是業務經驗的重要規則，透過檢定卻產生不顯著的結果時，可能是變數資料結構的問題，建議可透過資料轉換，變數切等、或分群方法來強化變數的解釋能力。 因此，變數篩選節點也可以用來檢視變數資料的狀況，提供變數是否需要進行資料處理的參考資訊。 圖一、Variable Selection node參數列 R-Square 各項主要參數選項的說明如下： ● 最大投入變數門檻值設定 (Maximum Variable Number) —設定投入變數的最大門檻值，預設值為3000。 ● 最小相關係數平方值 (Minimum R-Square) —預設值為0.005，若小於預設值則變數角色會被設定為拒絕(Reject)。 ● R2增益門檻設定 (Stop R-Square) —除了利用R2預設值進行變數篩選外，亦可以透過R2增益門檻設定，進行篩選調整，若投入變數的R2增益值低於預設值，該變數的變數角色亦會被設為拒絕，預設值為 0.0005. ● AOV16數值資料類別化 (Use AOV16 Variables) — 將數值資料類別化，透過Use AOV16 Variables 參數設定，將連續變數切分成16等分，AOV16 variables可以處理與目標變數的非線性關係，預設值為No. ● 類別變數水準值化簡 (Use Group Variables) — 運用 Use Group Variables 參數設定進行類別變數水準值化簡，依據投入變數與目標變數間的關係進行最適化簡，預設值為Yes. Chi-Square 各項主要參數選項的說明如下： ● 最小卡方門檻值設定 (Minimum Chi-Square) — 設定類別變數的最小卡方門檻值，預設值為3.84，因為P( chi-square統計量 > 3.84 ) = 0.05，也就是p-value設定在0.05，若分析變數卡方檢定結果其值小於3.84時，該變數的變數角色會被設為拒絕。 ● 連續變數類別化 (Number of Bins) — 將連續變數進行資料級距切割，已進行卡方分析，預設值為50。 Variable Selection node 變數篩選方法 Variable Selection node 變數篩選方法主要透過R2或卡方值的計算結果進行變數過濾。如果目標變數屬名目或順序尺度的變數，EM會產生一個 二元虛擬 的目標變數，而不是使用原來的順序或名義目標變量。在逐步回歸分析設定較低的進入門檻和排除門檻以避免排除可能適合的潛在影響變數。 SAS ... Read More

Feed: The Internet of Things on AWS – Official Blog. Author: Rodrigo Merino. Industrial IoT (IIoT) bridges the gap between industrial equipment and automation networks (usually called OT, Operations Technology) and Information Technology (IT). In IT, use of new technologies such as machine learning, cloud, mobile, and edge computing are becoming commonplace. IIoT brings machines, cloud computing, analytics, and people together to improve performance, productivity, and efficiency of industrial processes enabling customers to turn to IIoT applications for predictive quality and maintenance and to remotely monitor their operations from anywhere.But realizing the value of IIoT is challenging and usually there ... Read More

Feed: AWS Partner Network (APN) Blog. Author: Scott Kellish. By Scott Kellish, Partner Solutions Architects at AWSBy Basil Faruqui, Principal Solutions Marketing Manager at BMC Digital Business AutomationBy Joe Goldberg, Innovation Evangelist at BMCPredictive maintenance, which analyzes sensor data to predict equipment failures, has emerged as one of the most common business use cases of machine learning (ML) and the Internet of Things (IoT). To build and train ML models, you need data science expertise. If you’re going to run those ML models in production at scale, you need data engineering expertise to build a pipeline for data ingestion, storage, ... Read More

Feed: Featured Blog Posts - Data Science Central. Author: Ramesh. Predicting Smart Meters’ Next Non-Communicative Day Smart meters are used for billing of electricity, gas, water, and heat for residential, commercial, and production sites. The technological advancements in these smart meters are being driven by the demand from utility companies to create a smarter and more efficient grid aimed at reducing non-revenue losses. However, these smart meters have challenges to maintain and identifying problems. Identifying Faults/ Maintenance Challenges Smart meters go non-communicative for many reasons which include an excess of electric energy measurement, under voltage of battery, communication failure, timing ... Read More

Feed: R-bloggers. Author: n n R Analystatistics Swedenn. In my last post, I analysed how the level of education is affected by region, sex and year. In this post, I will continue with the same dataset but this time I will include age in the analysis. Please send suggestions for improvement of the analysis to [email protected]. First, define libraries and functions. library (tidyverse) ## -- Attaching packages -------------------------------------------------- tidyverse 1.3.0 -- ## v ggplot2 3.2.1 v purrr 0.3.3 ## v tibble 2.1.3 v dplyr 0.8.3 ## v tidyr 1.0.2 v stringr 1.4.0 ## v readr 1.3.1 v forcats 0.4.0 ## ... Read More

Feed: R-bloggers. Author: arthur charpentier. Let us get back on the Titanic dataset, 1 2 3 4 loc_fichier = "http://freakonometrics.free.fr/titanic.RData" download.file(loc_fichier, "titanic.RData") load("titanic.RData") base = base[!is.na(base$Age),] On consider two variables, the age x (the continuous one) and the survivor indicator y (the qualitative one) 1 2 X = base$Age Y = base$Survived It looks like the age might be a valid explanatory variable in the logistic regression, 1 2 3 4 5 6 7 8 9 10 11 12 13 14 summary(glm(Survived~Age,data=base,family=binomial))   Coefficients: Estimate Std. Error z value Pr(>|z|) (Intercept) -0.05672 0.17358 -0.327 0.7438 Age -0.01096 0.00533 -2.057 0.0397 ... Read More

Feed: Planet MySQL; Author: Valeriy Kravchuk; My weekdays are still busy even during this lockdown period, but weekend is a perfect time for yet another blog post about MySQL bugs I considered interesting. Very few followers read my posts on other topics anyway, especially if they have the "M....DB" word...So, here is a new list of optimizer, replication, InnoDB and few other bugs in GA versions of MySQL that I've subscribed to since the end of February, 2020:Bug #98675 - "LIMIT clause in derived table garbles EXPLAIN Note". It's not a big deal and in specific case presented by Øystein Grøvlen ... Read More

Feed: Planet PostgreSQL. Depending on your workload, the planning time can represent a significant part of the overal query procesing time. This is especially import in OLTP workload, but OLAP queries with numerous tables being joined and an aggressive configuration on the JOIN order search can also lead to hight planning time. Planning counters in pg_stat_statements Previously, pg_stat_statements was only keeping track of the execution part of a query processing: the number of execution, cumulated time, but also minimum, maximum, mean and also the standard deviation. With PostgreSQL 13, you’ll also have those metrics for the planification part! commit 17e03282241c6ac58a714eb0c3b6a8018cf6167a ... Read More